Mailman 3 February 2007 - Python-ideas

fixing mutable default argument values
by Chris Rebert 02 May '07

02 May '07

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)

9 25

Re: [Python-ideas] [Python-Dev] Making builtins more efficient
by Greg Ewing 20 Apr '07

20 Apr '07

Steven Elliott wrote: > What I have in mind may be close to what you are suggesting above. My idea is somewhat more uniform and general than that. For the module dict, you use a special mapping type that allows selected items to be accessed by an index as well as a name. The set of such names is determined when the module's code is compiled -- it's simply the names used in that module to refer to globals or builtins. The first time a given builtin is referenced in the module, it will be unbound in the module dict, so it is looked up in the usual way and then written into the module dict, so it can subsequently be retrieved by index. The advantages of this scheme over yours are that it speeds access to module-level names as well as builtins, and it doesn't require the compiler to have knowledge of a predefined set of names. It does entail a slight semantic change, as changes made to a builtin won't be seen by a module that has already used that builtin for the first time. But from what Guido has said before, it seems he is willing to accept a change like that if it will help. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing(a)canterbury.ac.nz +--------------------------------------+

2 1

Re: [Python-ideas] [Python-Dev] bool conversion wart?
by Greg Ewing 03 Mar '07

03 Mar '07

Ron Adam wrote: > Ok, so what if... instead of bool being a type, it be a keyword that is > just a nicer way to spell 'not not'? The main benefit of having a bool type is so that its values display as "True" and "False" rather than whatever surrogate values you happen to have used to represent true and false. The fact that bool() can be used canonicalise a truth value is secondary -- the same thing could just as well be achieved by a plain function (as it was before bool became a type). It's an extremely rare thing to need to do in any case. If any change is to be made to bool in 3.0, my preference would be to keep it but make it a separate type. > Are 'and' and 'or' going to be changed in 3.0 to return bool types too? I very much hope not! Their current behaviour is very useful, and I would hate to lose it. -- Greg

4 11

immutable classes [was: pre-PEP: Default Argument Expressions]
by Jim Jewett 28 Feb '07

28 Feb '07

I have added python-ideas to the Cc list, and suggest removing python-3000 from additional replies. BJörn Lindqvist gave an example explaining why he might want to re-evaluate mutable default arguments. It still looks like like buggy code, but it isn't the error I was expecting -- and I think it comes from the difficulty of declaring something immutable. On 2/15/07, BJörn Lindqvist <bjourne(a)gmail.com> wrote: > On 2/15/07, Jim Jewett <jimjjewett(a)gmail.com> wrote: > > Then are there *any* good use cases for [non-persistent mutable defaults] > > (1) Not really (treated as) mutable. ==> Doesn't care > > >>> def f(extra_settings={}) ... > > usually doesn't modify or even store extra_settings; ... > That is dangerous code. Sooner or later someone will modify the > extra_settings dict. How? >>> f.func_defaults[0]['key']=value may be misguided, but it probably isn't an accident. BJörn's example does store the mutable directly, but it makes a bit more sense because it looks like a complex object rather than just a mapping. > class Vector: > def __init__(self, x, y, z): > self.x = x > self.y = y > self.z = z > class Ray: > def __init__(self, direction, origin = Vector(0, 0, 0)): > self.direction = direction > self.origin = origin > > ray1 = Ray(Vector(0, 0, -1)) > ray2 = Ray(Vector(0, 0, 1)) > ray3 = Ray(Vector(-1, 0, 0), Vector(2, 3, 4)) > The above code looks quite nice, but is wrong. Why is vector mutable? Is the real problem that it is too hard to declare objects or attributes immutable? My solution is below, but I'll grant that it isn't as straightforward as I would have liked. Is this something that could be solved with a recipe, or a factory to make immutable classes? >>> class Vector3D(tuple): ... def __new__(self, x, y, z): ... return super(Vector3D, self).__new__(self, (x, y, z)) ... x=property(lambda self: self[0]) ... y=property(lambda self: self[1]) ... z=property(lambda self: self[2]) -jJ

4 8

A Pythonic replacement for ConfigParser?
by Larry Hastings 25 Feb '07

25 Feb '07

Ah, yes, ConfigParser, the red-headed stepchild of Python serialization. It seems ConfigParser has been around for a /very/ long time... and it shows. In particular, its interface is pretty clunky when viewed with a modern Python programmer's eye. I propose hacking up ConfigParser so it has a more Pythonic interface--specifically, so it acts more-or-less like a two-level nested dict. I'll call the rewritten module "configdict" for now. Here's the full list of what I have in mind: * a configdict object behaves like a dict, but can only contain configdict.section objects * a configdict.section object behaves like a dict, but can only contain strings * obviously, configdict and configdict.section would be new-style classes * write the initial implementation as a wrapper around RawConfigParser * "defaults" argument to constructor -> configdict.update() or configdict[section].update() * sections() -> configdict.iternames() * add_section(s) -> configdict[s] = section * has_section(s) -> s in configdict * readfp(fp) -> configdict.read(fp) * add a new function parse(s) which parses s as if it were a file (probably just wraps in stringio and calls read()) * get(section, option) -> configdict[section][option] or configdict[section].get(option) * items(section) -> configdict[section].iteritems() * write(fp) -> unchanged * remove_option(section, option) -> del configdict[section][option] * remove_section(section) -> del configdict[section] * drop read(filenames), getint(), getboolean() * not sure how I feel about optionxform(), could keep it, could drop it * drop the spooky "magic interpolation feature" entirely (explicit, not implicit) Worth pursuing? Cheers, /larry/

3 2

Re: [Python-ideas] [Python-Dev] Making builtins more efficient
by Greg Ewing 21 Feb '07

21 Feb '07

Giovanni Bajo wrote: > Are you aware of this patch, which is still awaiting review? > https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_… That scheme seems unnecessarily complicated to me. I don't see a need to mess around with timestamps, except to preserve the ability to change builtins on the fly, and I don't think it's worth going to great lengths to preserve that. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing(a)canterbury.ac.nz +--------------------------------------+

1 0

local variable access : __getlocal__ and __setlocal__
by Pierre-Yves Martin 18 Feb '07

18 Feb '07

Currently it's possible to customize attribute access via methods : __getattribute__(self, name) __getattr__(self, name) __setattr__(self, name, value) but it is not possible to customize local variable access. It would be useful for example to allow implementation of a local constant or any other on-access/on-modification behavior. The function should could be: __getlocal__(name) __setlocal__(name, value) By default they are just access to the local dictionnary for reading and writting (AFAIK the only current way to access this dictionnary is using the locals() built-in). Here is a sample code of typical usage : ####################### #Sample 1 : Tracing local access old_getlocal = __getlocal__ old_setlocal = __setlocal__ #redefine the built-ins def __getlocal__(name): print 'accessing %s' % name return old_getlocal(name) def __setlocal__(name, value): print 'setting %s with value %s' % (name, value) old_setlocal(name, value) a = 1 b = 2 c = 3 #setting b and accessing a b = a + 1 ################### #Sample 2 : Constants old_getlocal = __getlocal__ old_setlocal = __setlocal__ class ConstantError(Exception):pass constants = [] #redefine the built-ins def __setlocal__(name, value): if old_getlocal(name, value) in constants: raise ConstantError() else: old_setlocal(name, value) a = 1 b = 2 c = 3 constants = [b, c] a = 4 #OK b = 5 #IMPOSSIBLE constant.remove(c) c = 6 #OK I found such approach more flexible than introducing a 'const' in python (not a good thing I think). And it enable many other customizations... Note : the fact that in my example I DO NOT access to any kind of writable local dictionary! This on purpose, the only way to access this dictionary should be : locals(), __getlocal__() and __setlocal__() built-ins. Pierre-Yves Martin pym<dot>aldebaran< a t >gmail<dot>com

3 2

Pictograms in Python?
by Roman Susi 18 Feb '07

18 Feb '07

Hi! I have recently came across this part of Unicode spectrum: http://www.nttdocomo.co.jp/english/service/imode/make/content/pictograph/li… While it is understood that the visual representation depends on many things and doesn't depend on Python that much, it could perhaps be useful to have unicodedata contain at least translations, so that unicodedata.name(unichr(0xE63F)) be something like: "PICTOGRAM CLOUDY" Still, I donno how standard is that range of Unicode characters, but if Unicode really has those their support could be funny... Hopefully, BDFL will 0xE70B that 0xE6F0 See you 0xE6B7, Roman

2 1

Packages and Import
by Ron Adam 12 Feb '07

12 Feb '07

After exploring this a bit further on comp.lang.python, I was able to organize these ideas better. The more I thought about it, the more '+'s I found, and about the only '-'s I can think of is the work required to actually make a patch to do it. It's also good to keep in mind that since most people still rely on the old relative import behavior, most people have not run into some of the issues I mention here. But they will at some point. I did mean to keep this short, but clarity won out. (At least it's clear to me, but that's an entirely subjective opinion on my part.) Maybe someone will adopt this and make a real PEP out of it. :-) Cheers, Ron PROPOSAL ======== Make pythons concept of a package, (currently an informal type), be stronger than that of the underlying file system search path and directory structure. Where the following hold true in python 3.X, or when absolute_import behavior is imported from __future__ in python 2.X: (1) Python first determines if a module or package is part of a package and then runs that module or package in the context of the package they belong to. (see items below) (2) import this_package.module import this_package.sub_package If this_package is the same name as the current package, then do not look on sys.path. Use the location of this_package. (3) import other_package.module import other_package.sub_package If other_package is a different name from the current package (this_package), then do not look in this_package and exclude searches in sys.path locations that are inside this_package including the current directory. (4) import module import package Module and package are not in a package, so don't look in any packages, even this one or sys.path locations inside of packages. (5) For behaviors other than these, like when you do actually want to run a module belonging to a package in a different context, a mechanism such as a command line switch, or a settable import attribute should be used. MOTIVATION ========== (A) Added reliability. There will be much less chance of errors (silent or otherwise) due to path/import conflicts which are sometimes difficult to diagnose. There may also be some added security benefits as well because it would much harder for someone to create a same named module or package and insert it by putting it on the path. Or by altering sys.path to do the same. [*] [* - If this can happen there are probably more serious security issues, but not everyone has the most secure setup, so this point is still probably a good point. General reliable execution of modules is the first concern, this may be a side benefit of that.] (B) Reduce the need for special checks and editing sys.path. Currently some authors have edit sys.path or do special if os.path.exists() checks to ensure proper operations in some situations such as running tests. These suggestions would reduce the need for such special testing and modifications. (D) Easier editing and testing. While you are editing modules in a package, you could then run the module directly (as you can with old style relative imports) and still get the correct package-relative behavior instead of something else. (like an exception or wrong output). Many editors support running the file being edited, including idle. It's also can be difficult to write scripts for the editors to determine the correct context to run a module in. (E) Consistency with from ... import ... relative imports. A relative import also needs to find it's home package(s). These suggestions are consistent with relative import needs and would also enable relative imports to work if a module is run directly from and editor (like idle) while editing it. [*] [* - Consistency isn't a major issue, but it's nice to have.] (F) It would make things much easier for me. ;-) (Insert "Me Too's" here.) DISCUSSION ========== (I) Python has a certain minimalist quality where it tries to do a lot with a minimum amount of resources. (Which I generally love.) But in the case of packages, that might not be the best thing. It is not difficult for python to detect if a module is located in a package. With the move to explicit absolute/relative imports, it would make since if Python also were a little smarter in this area. Packages are becoming used more often and so it may also be useful to formalize the concept of a package in a stronger way. (II) Many of the problems associated with imports are a side effect of using the OS's directory structure to represent a python "package" structure. This creates some external dependence on the operating system that can effect how python programs run. Some of these issues include: - Importing the wrong package or module. - Getting an error due to a package or module not being found. - Getting an error due to a package not being loaded or initialized first. - Having to run modules or packages within a very specific OS file context. - Needing a package location to be in the systems search path. By making the concept of a package have priority over the OS's search path and directory structure, the dependence on the OS's environment is lessoned and it would insure a module runs in the correct context or give meaningful exceptions in more cases than presently. (III) If a package was represented as a combined single file. Then the working directory would always be the package directory. The suggestions presented here would have that effect and also reduce or eliminate most if not all of these problem situations. (IV) The suggested changes would change the precise meaning of an absolute import. Given the following example of an un-dotted import: >>> import foo The current meaning is: "A module or package that is located in sys.path or the current directory". But maybe a narrower interpretation of "absolute import" would be better: "A module or package found in a specific package." I believe that this latter definition is what most people will think of while editing their programs. When dotted imports are used, the left most part of the name is always a top level package or module in this case. (V) Requirements to be on the search path. It is quite reasonable to have python modules and packages not in the search path. Conversely it is not reasonable to require all python modules or packages to be in locations listed in sys.path. While this isn't a true requirement, it is often put forth as a solution to some of the problems that occur with respect to imports. (VI) Clearer errors messages. In cases where a wrong module or package is imported you often get attribute exceptions further in the code. These changes would move that up to the import statement because the wrong module would not be imported. (VII) Setting a __package__ attribute. Would it be a good idea to have a simple way for modules to determine parent packages, and their absolute locations? Python could set these when it starts or imports a module. That may make it easier to write alternate importers that are package aware. PROBLEMS AND ISSUES: - Someone needs to make it happen. I really can't think of any more than that. But I'm sure there are some as most things like this are usually a trade off of something.

5 27

New syntax for 'dynamic' attribute access
by Ben North 11 Feb '07

11 Feb '07

Thanks for the responses on this. In general: Guido van Rossum: > I think you should submit this to the > PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- > there's no benefit that I see of waiting until 3.0. Greg Falcon: > Wow! I have to say this is a compelling idea. Josiah Carlson: > My only concern with your propsed change is your draft implementation. and on the syntax in particular: Guido van Rossum: > I've thought of the same syntax. Greg Falcon: > The syntax is a bit foreign looking, but [...] I feel like I could > learn to like it anyway. Mostly positive, then, as far as the general idea and the syntax goes. On the two-argument form, Greg Falcon wrote: > > x = y.('foo_%d' % n, None) > > This is the one bit I really don't like. y.('foobar') is arguably a > natural extension to y.foobar, but y.('foobar', None) isn't analogous > to any current syntax, and the multiple arguments make it look even > more dangerously like a call. > > In Python, you already have to be explicit when you're worried if the > thing you're accessing might not be there. I was definitely in two minds about whether the extension to the two-argument form was a win or a loss. It does increase the power of the new syntax, but it is certainly rather clumsy-looking. I'd be happy to prepare a patch with just the one-argument version if the consensus is that the balance is that way. Josiah Carlson: > My only concern with your propsed change is your draft > implementation. [...] > > Specifically, your changes to ceval.c and the compiler may have been > easier to implement, but it may negatively affect general Python > performance. Have you run a recent pystone before and after the changes? I hadn't done so, no, but have now tried some tests. In fact, there is some evidence of a slight negative effect on the general performance. On my laptop, repeated pystone runs with 100,000 loops are quite variable but there might be a performance penalty of around 1% with the new code. I'm a bit puzzled by this because of course the new opcodes are never invoked in the pystone code --- does a switch statement become slower the more cases there are? (I tried grouping the cases with the new opcodes all together at the end of the switch, but the noisiness of the pystone results was such that I couldn't tell whether this helped.) Or is it just that the core bytecode-interpretation loop is bigger so has worse processor cache performance? On the other hand, getting/setting dynamic attributes seems to be about 40--45% faster with the new syntax, probably because you avoid the lookup of 'getattr' and the overhead of the function call. Anyway, I'll experiment with the other implementation suggestions made by Josiah, and in the meantime summarise the discussion so far to python-dev as suggested by Guido. Thanks, Ben.

3 3