Mailman 3 - Python-ideas

Re: [Python-ideas] fixing mutable default argument values
by Jan Kanis Feb. 7, 2007

Feb. 7, 2007

Other non-python stuff is taking my time, but I'll reply to this anyway. Better late than never (in this case). On Thu, 01 Feb 2007 16:19:00 +0100, Lucio Torre <lucio.torre(a)gmail.com> wrote: > On 1/31/07, Jan Kanis <jan.kanis(a)phil.uu.nl> wrote: >> > >> > Isnt the other argument the fact that always re evaluating will bring >> > similar gotchas? >> >> IMO, it won't. Caching will have to be implemented in a different way, >> … [View More]but >> that wouldn't really be a gotcha (except for python 2.x programmers who >> didn't read the "what's new in python 3.0") >> > > Maybe is better to annoy newbies that didnt read the python FAQ than > seasoned python 2.x programmers! :) That statement I can agree with, but it isn't really applicable here. The applicable statement reads: "Maybe it's better to annoy newbies that didn't read the python FAQ than python 2.x programmers who don't know anything about 3.0 and assume it's just another new version of python." And I don't agree with that one. :) >> If the >> function depends on other state I'd imagine that most of the time you'd >> want a change in state to be reflected in the functions default value. >> >> example: >> def foo(x = exec_sql("SELECT someval FROM defaults_table WHERE >> some_expr")): >> # do something >> > > i was thinking more in the lines of the case when you dont want the > default to change because something has changed. Then don't make the default depend on something you're going to change. Well, what this is really about is about closure vars changing when the outer scope changes them instead of the value of the closure vars being captured by the new lexical scope. This has been proposed before (to change this behaviour on closure variables in general) and rejected. (though I am in favour of it) Python is a late binding language in just about everything, and that is considered to be a good thing. The point you're making is exactly the early-vs-late binding discussion. > I still think this would be optimizing towards the case where you > should put the code in the body. What if i want a static default > parameter, never to change, but i want to compute it at compile time? > Should i need to use extra syntax for that? Isnt this case more common > than the others? Something like: > > def foo( pixels = (640*480) ): pass > > Should overriding '*' change that default value? (can you do that in > python3k? i think you cant in 2.x) I'm not really sure if you can either. Anyway, the only cases I can think of when this makes sense is when there's a bug in '*' and you want to patch it, or if you're overloading '*' to work with other values, without changing the meaning for ints. In both of these cases it would be a good resp. neutral thing to late-bind '*' > On the other hand, i think that learning about compile time and run > time is a great thing and should be encouraged. Yup. But let's not fill the language with gotchas for the newbies to find and learn from :) >> > And having code that is invoked every time a function is called.. >> > shouldnt the place for this be the function body? >> >> The only reason to answer 'yes' is because it is what you're currently >> used to. > > Im not only used to it, i like it :) You'll get to like the new way :) >> If I see a function with a default argument of [], I expect it to >> use an empty list when I call it without the argument. >> > > That is true. I hate that. But i still like it better than the other > cases. This can be learnt easily and gives you the power to do > everything the other solutions have. The other solutions require more > syntax and more learning to achieve the same. Id love to find a > solution for this, but this options dont seem like one. Please note that I'm not in favour of any of the possible new syntaxes, I am only in favour of changing the semantics without new syntax (cf the pre-pep). Since many newbie python programmers run into this, it seems that it is easier to learn that python evaluates the default expressions at calltime, because that is what the rest of python makes the newbies expect when they first try it. So for newbies, there would actually be less learning involved. You will have to remember this change when you start using py 3.0, but this won't be the only thing to remember. And there are still several other ways to make some python code evaluate the expression at deftime. >> Having code that is eval'd when the function is defined, shouldn't the >> place for this be outside the function definition? > > i dont know, "def fname (...):" is executed when the code block is > executed (creating the function object), and the body is executed when > the function is called. So having a statement that has both compile > time and run time effects is kind of messy. As the body is part of the def statement, the def is already doing exactly that (being both compile/def time and run/call time). Consider the ...'s to be part of the function body in the new semantics. Instead of part of the code in a def being executed at deftime and part at calltime, everything is just run at calltime. There are some pretty large warnings in the docs about this distinction currently, they can be taken out. > I hope im still making sense and not playing out of my league, but > thats just my 2c. > > Lucio. - Jan [View Less]

2 1

proto-PEP: Fixing Non-constant Default Arguments
by Chris Rebert Feb. 2, 2007

Feb. 2, 2007

The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default … [View More]arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=<baz>): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/ [View Less]

10 42

Re: [Python-ideas] File system API
by Tal Einat Feb. 2, 2007

Feb. 2, 2007

On 2/2/07, Eduardo EdCrypt O. Padoan <eopadoan(a)altavix.com> wrote: > I think that there aready exists a proposal for an Abstract FS Layer > for Python somewere. > I haven't really seen a proposal for such a thing. itools.vfs is the only actualy file-system API, or file-system abstraction layer, I know of. It is going more or less in the directions I am currently thinking of. But the fact is that it has not gained popularity makes me think that maybe the direction needs to … [View More]

3 2

Re: [Python-ideas] File system API
by Eduardo "EdCrypt" O. Padoan Feb. 2, 2007

Feb. 2, 2007

> > I think that there aready exists a proposal for an Abstract FS Layer > > for Python somewere. > > I haven't been able to find any mention of it. Maybe you could point me in > the right direction? > > - Tal > http://wiki.python.org/moin/CodingProjectIdeas/FileSystemVirtualization Another related reference is the recently announced UniPath module: http://sluggo.scrapping.cc/python/unipath/Unipath-current/README.html -- EduardoOPadoan (eopadoan->altavix::… [View More]

1 0

Re: [Python-ideas] File system API
by Eduardo "EdCrypt" O. Padoan Feb. 2, 2007

Feb. 2, 2007

I think that there aready exists a proposal for an Abstract FS Layer for Python somewere. -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com

1 0

File system API
by Tal Einat Feb. 2, 2007

Feb. 2, 2007

I'm proposing a general file-system API (for Python of course). Basically, that's it. I've been wondering how many thousands of times this has been discussed, but an hour's worth of searching the net actually gave very little on this subject. Most other proposals I've seen started far, far further into developing the idea, and I think it's worth giving it a fresh start from the beginning. Also, most proposals centered around either creating efficient fake file-systems, simplifying working … [View More]

1 0

Import and '..', '../..' in serach path.
by Ron Adam Jan. 31, 2007

Jan. 31, 2007

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package. Module in package: import sys sys.path = ['..'] + sys.path import package.module # Imports module in "this!" package. Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of … [View More]the path. Module in sub-package: import sys sys.path = ['../..'] + sys.path import package.subpackage.module # finds "self" (subpackage) reliably. By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path. Adding this tip to the documentation some where would be nice. (providing there is no major surprising side effects.) Of course I may have missed some obvious way to do this. If so, it wasn't in an obvious place to be found. I looked. ;-) ---------------------------------- It might be useful to have a built-in function to do this. A function could also check for __init__ files and raise errors if they are missing. set_package_name(dotted.name) # Replaces import sys & path modification Where dotted.name is the full package + sub-package name the current module is located in. The function would search upwards to get the root package directory and add that to the *front* of sys.path. Module in package: set_package_name('package') # Add parent directory to front of sys.path import packagename.module # Finds module in "this!" package reliably. Module in subpackage: set_package_name('package.subpackage') import package.subpackage.module # Finds "self" (subpackage) reliably. ---------------------------------- It may also be able to modify the import behavior to allow relative imports to work when the module is run as script. set_package_name('package') from . import module1 # Imports modules from "this" package. from . import module2 Currently an exception is raised you try to run a module with relative references as a script. ValueError: Attempted relative import in non-package I think it is very handy to be able to run tests as scripts and keep them in a sub-package. Especially while I'm writing them. Another benefit of using relative imports with an absolute specified package name, is if you rename a package or relocate a submodule, you only have one line to change. Cheers, Ron [View Less]

2 7

Mutable default arguments - another approach
by Piotr Duda Jan. 31, 2007

Jan. 31, 2007

I think that this problem can be solved by the following change of default argument behavior: default arguments are evaluated in definition time (like in current implementation), but right after being evaluated, result object is checked if it's mutable (for example by checking of presence __copy__ special method or being instance of built in (sub)class list/dict/set), if object is mutable, argument is marked by COPY_DEF_ARG flag. There are two reasons for this check being done there: 1. … [View More]

6 7

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by Eduardo "EdCrypt" O. Padoan Jan. 30, 2007

Jan. 30, 2007

On 1/30/07, Eduardo EdCrypt O. Padoan <eopadoan(a)altavix.com> wrote: > > If I have time and figure out the right regexes I'll try and come up with > > some more numbers on the entire stdlib, and the ammount of uses of =None. > > > > Some uses of the spam=[] and ham=None in Python projects, including > Python itself: > > http://www.google.com/codesearch?q=def.*%5C(.*%3D%5C%5B%5C%5D.*%5C)%3A%20la… > http://www.google.com/codesearch?hl=en&lr=&q=… [View More]

1 0

Re: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

there really is no need to have something like that built into the language. most default arguments are immutable anyhow, and i'd guess most of the mutable defaults can be addressed with the suggested @copydefaults decorator. as for uncopyable or stateful defaults (dev = sys.stdout), which require reevaluation, you can just use this modified version of my first suggested decorator: >>> def reeval(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... … [View More]

1 0