Mailman 3 May 2007 - Python-ideas

Ideas towards GIL removal
by Greg Ewing 18 Jul '08

18 Jul '08

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". Any refcount operation on a permanent object would be a no-op, so no locking would be needed. This would also have the benefit of eliminating any need to write to the object's memory at all when it's only being read. 2) Objects owned by a thread Python code creates and destroys temporary objects at a high rate -- stack frames, argument tuples, intermediate results, etc. If the code is executed by a thread, those objects are rarely if ever seen outside of that thread. It would be beneficial if refcount operations on such objects could be carried out by the thread that created them without locking. To achieve this, two extra fields could be added to the object header: an "owning thread id" and a "local reference count". (The existing refcount field will be called the "global reference count" in what follows.) An object created by a thread has its owning thread id set to that thread. When adjusting an object's refcount, if the current thread is the object's owning thread, the local refcount is updated without locking. If the object has no owning thread, or belongs to a different thread, the object is locked and the global refcount is updated. The object is considered garbage only when both refcounts drop to zero. Thus, after a decref, both refcounts would need to be checked to see if they are zero. When decrementing the local refcount and it reaches zero, the global refcount can be checked without locking, since a zero will never be written to it until it truly has zero non-local references remaining. I suspect that these two strategies together would eliminate a very large proportion of refcount-related activities requiring locking, perhaps to the point where those remaining are infrequent enough to make GIL removal practical. -- Greg

8 19

Attribute Docstrings and Annotations
by Ali Sabil 01 Jun '07

01 Jun '07

Hello all, I was looking into function annotations, that are to be introduced in Python3k, and I found this old message sent to this mailing list : http://mail.python.org/pipermail/python-ideas/2007-January/000037.html I would like to restart the discussion of attribute annotation, because in my opinion it can be a very powerful feature. I personally think about using it for SOAP message serialization, or any kind of XML serialization, the Idea would be to annotate various object attribues to be either marshaled as XML Elements or XML Attributes of the current Node that reflects the Object. Thank you, -- Ali

2 1

About list comprehension syntax
by Arnaud Delobelle 01 Jun '07

01 Jun '07

Hi List comprehensions (and generator expressions) come in two 'flavours' at the moment: (1) [f(x) for x in L], which stands for map(f, L). Let's call this a 'map comprehension' (2) [f(x) for x in L if p(x)], which stands for map(f, filter(p, L)). Let's call this a 'map-filter comprehension'. Now if one wants to write simply filter(p, L) as a list comprehension, one has to write: (3) [x for x in L if p(x)]. This could be called a 'filter comprehension'. the 'x for x in L' is not very nice IMHO, but it is often handy to use such expressions over 'filter(...)', eg building the sublist of a given list consisting of all the items of a given type could be written as: filter(lambda x: isinstance(x, FilteringType), heterogeneous_list) or: [x for x in heterogenous_list if isinstance(x, FilteringType)] I still prefer the list comprehension over the lambda/filter combination, but neither feels very satisfying (to me :) (not that one cannot use partial in the filter version) Why not just drop the 'x for' at the start of a 'filter comprehension' (or generator expression)? Thus (3) could be written more simply as: (3') [x in L if p(x)] This is consistent with common mathematical notation: * { f(x) | x \in L } means the set of all f(x) for x in L * { f(x) | x \in L, p(x) } means the set of all f(x) for x in L satisfying predicate p. * { x \in L | p(x) } means the set of all x in L satisfying predicate p. -- Arnaud

7 8

proposal to add rowexpr as a keyword
by Steve Howell 31 May '07

31 May '07

ABSTRACT: I propose the following syntax: rowexpr: convert_to_euros(salary) > 50000 and deptno = 10 Rowexpr returns a function object that can be called with one argument, row, and the items "salary" and "deptno" are used to derefence row. RATIONALE: A lot of time people write Python code that works with other small interpreters, such as SQL engines, math libraries, etc., that have an expression syntax of their own. To use SQL as an example, I may want my SQL engine to use this Python expression: where convert_to_euros(salary) > 50000 and deptno = 10 To interact with SQL engines, I could imagine code like the following: sql(..., lambda row: convert_to_euros(row['salary'] > 50000 and row['deptno'] = 10) # not concise sql(..., sql_parse('convert_to_euros(salary) > 50000 and deptno = 10') # requires sophisticated library sql(..., [and_op, [ge_op(func_call_op(convert_to_euros, 'salary'), 50000), eq_opt(itemgetter_op('deptno'), 10)]) # UGLY!!! SOLUTION: Let the Python interpreter build expression objects for you. You write: rowexpr: convert_to_euros(salary) > 50000 and deptno = 10 Python translates that into the same bytecode as: lambda row: convert_to_euros(row['salary'] > 50000 and row['deptno'] == 10 Benefits: 1) Allows smaller code. 2) Having Python translate the rowexpr creates an opportunity for better error messages, catching more things at startup time, etc. 3) In the particular case of SQL, it enable the development methodology that you might start off by writing code that processes data in memory, but with mostly SQL-like syntax, and then switch over to a real SQL database as scalability becomes a bigger concern. COMMON OBJECTIONS: 1) More syntax, of course. 2) Folks used to SQL would still need to adjust to differences between Python expressions (==) and SQL (=). 3) In the case of SQL, people may already be happy enough with current tools, and of course, interpreters external to Python can cache expressions, etc. OPEN QUESTIONS: Would a rowexpr be limited to dictionaries, or could it also try to access object attributes? ____________________________________________________________________________________Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=list&sid=396545469

9 27

Idea: The DSL operator.
by Talin 31 May '07

31 May '07

This is in response to the various suggestions for embedding SQL syntax in Python and various other suggestions made over the years. Note that this suggestion is only a starting point for discussion, I don't have any illusion that the idea as presented here would have strong support. I'm fully aware that the actual proposal given below looks "butt-ugly", but maybe someone can invent something better. Background: Python is frequently used as a foundation for Domain Specific Languages (DSLs). Examples include SQLObject, SCons, and many others. These are all examples of a mini-language which is constructed on top of the Python syntax, using operator overloading. The DSL is often embedded within Python code (as in the case of SQLObject), and can be invoked programmatically from regular (non-DSL) Python code. However, there are some limitations on what can be expressed. For example, in current interpreters, it is not possible to override the 'and' and 'or' operator. While there currently is a PEP that proposes adding this capability, there is some disagreement as to how much this would impact the efficiency of the non-override case. The reason is because in the non-override case, the compiler is able to take advantage of the "shortcut" properties of these operators; If the operators can be overridden (which is a post-compiler operation), then the compiler would no longer be able to make these optimization assumptions. A similar limitation in DSLs is the treatment of unbound identifiers. Identifiers which have already been bound to objects are not a problem, since you can simply override the various operators of those objects. However, for identifiers which have not yet been assigned to a variable, a number of ugly hacks are required. Typically what is done is to have a set of "special" objects which act as placeholders for unbound variable names, for example _x, _y, _z, and so on. Unfortunately, this requires the person writing the DSL expression to pre-declare unbound variables, which is exactly the opposite of how regular Python works. Another method that is sometimes used is to declare unbound variables as attributes of some general "unbound variable" object, so you can say something like Query.x, Query.y, and so forth. But again, this is rather wordy and robs the DSL of its ability to concisely express a particular semantic intent, which is the whole point of DSLs in the first place. Another limitation of Python's ability to represent DSLs is that you cannot override the assignment operator. My idea is to alleviate these problems by giving DSLs some help at the parsing level. By informing the parser that a given expression is a DSL, we can relax some of the normal assumptions of the Python syntax, and instead require the DSL interpreter to handle them, which allows the DSL to have greater flexibility. Two things we want to avoid: First, we want to avoid "programmable syntax", meaning that we cannot define DSL-specific semantics at parse time. The reason for this is that DSLs are generally defined in their own specific imported modules, and the Python parser has no knowledge of imports, and therefore has no access to the DSL's syntax rules. Secondly, we want to avoid conflicts between multiple DSLs. So for example, if I am using two different DSLs in the same module, we can't have two different meanings for "and", for example. The solution to both of these concerns is to say that the parser's support for DSLs is completely generic - in other words, we can tell the parser that a given expression is a DSL, but we can tell it nothing specific about that *particular* DSL. Instead, it is up to the DSL interpreter - which operates at runtime, not parse time - to take up where the parser left off. This is similar to PJE's suggestion of a way to "quote" Python syntax, returning an AST. In such a scheme, the Python parser doesn't completely compile the quoted expression to Python byte code, but instead produces an AST (or AST-like data structure which might not exactly be an AST), which is then interpreted by the DSL. So far this all seems reasonable enough to my ear :) Next is the part where I go mad. (As my friend Sara said to me recently, "I'm not a mad scientist - I'm a mad natural philosopher.") For purposes of dicussion I am going to introduce the token '??' (two consecutive question marks) as the "DSL operator", in other words a signal to the parser that a given expression is a DSL. I'm sure that there could be a long and intense bikeshed discussion debating the specific operator syntax, but for now I'll choose ?? since it's unambiguously parsable. The '??' operator can be used in one of three ways. First, you can declare an entire expression as a DSL: a = ??(c + d) The parser will take all of the tokens inside the parenthesized group and return them as an AST, it will not attempt to generate code to evaluate any of the operators within the group. There's an extra bit of syntactic sugar: If the DSL expression is the only argument to a function, you can combine the function call and the DSL quote: func??(c+d) Meaning "pass the AST for 'c + d' to the function 'func'". For most DSLs, the 'func' will actually be the name of the DSL, so the typical use pattern will be 'name_of_dsl??(expr)'. The second use is to quote an identifier: a = ??c + d Now, you would think that this would quote the variable 'c' and attempt to add it to 'd', but this is not the way it works. Instead, the ?? operator invokes a kind of operator overloading that happens at the parser level, meaning that any attempt to create an expression that operates on a quoted variable also gets quoted. This only affects binary and unary operators, not functions calls, array references or assignment. So in this case, the entire expression "??c + d" gets quoted. The third way a DSL operator can be used is to quote an operator: a = c ??= d Again, this allows us to generate the AST "c = d" and assign it to the variable "a". (I recognize that these last two syntactical variations would make the interpretation of regular Python ASTs significantly more complex, which is a serious drawback. Part of the motivation for both of these variations is to deal with the 'and', 'or' and '=' problem, by overriding the Python compiler's normal treatment of these operators. However, it may be that the first form alone is sufficient, which would be relatively easy to handle in the Python compiler I think - you would simply have the AST.c module look for a DSL operator, and don't call the normal AST processing steps for arguments to that operator.) Some examples of DSLs using the operator: An SQL query as a DSL: query = SQL??(name == "John" and age > 25) result_iter = query.execute() Syntax for an algebraic solver: @arity??(x + x) def reduce(x): return 2 * x @arity??(x + 0) def reduce(x): return x @arity??(x * 0) def reduce(x): return 0 @arity??(x * 1) def reduce(x): return x @arity??(x * a + y * a) def reduce(x): return (x+y)*a # (and so on for all the other reduce() rules) print(reduce??((a+b*3) + (a+b*3))) >>> a*2+b*6 (In the above, we define a function 'reduce' that is similar to a generic function, except that the dispatch logic is based on the form of the AST being passed as an argument, using a one-way unification algorithm to do argument pattern matching.) I realize that there are some aspects of this proposal that may seem silly, and that's true, and the only reason I'm even suggesting this at all is that much of what I'm writing here is already been suggested or at least implied by previous discussions. Flame on! -- -- Talin

3 3

Make a CPAN like system for Py3K
by Leonardo Santagada 31 May '07

31 May '07

I know about cheeseshop and I know about setuptools. But how about all this be standard tools on python 3.0, so we can have something like CPAN but better. With setuptools being default on py3k you can already easy_install everything, ok but then we can do even more. When the user run a python script, if it requires a version of a lib that he doesn't have it could download it and install in his home folder (something in the lines of ~/pythonlibs). This will make it much easier to distribute python programs. I think this is somewhat like what java webstart does. What do you guys think? Also why no one is talking about having a tool like setuptools in the stdlib? -- Leonardo Santagada "If it looks like a duck, and quacks like a duck, we have at least to consider the possibility that we have a small aquatic bird of the family anatidae on our hands." - Douglas Adams

2 1

positional only arguments decorator
by Steven Bethard 30 May '07

30 May '07

Ok, looks like there's not much chance of agreeing on a syntax, so here's a decorator that covers the two use cases I know of:: * matching the signature of dict() and dict.update() * allowing arguments to have their names changed without worrying about backwards compatibility (e.g. ``def func(sequence)`` changing to ``def func(iterable)``) I've pasted the docstring and code below. The docstring should give you a pretty good idea of the functionality. Let me know if you have use cases this wouldn't cover. """ Functions declared ad @positional_only prohibit any arguments from being passed in as keyword arguments:: >>> @positional_only ... def foo(a, b=None, *args): ... return a, b, args ... >>> foo(1, 2, 3) (1, 2, (3,)) >>> foo(a=1, b=2) Traceback (most recent call last): ... TypeError: foo() does not allow keyword arguments Note that you can't use **kwargs arguments with a @positional_only function:: >>> @positional_only ... def foo(a, b=None, **kwargs): ... pass ... Traceback (most recent call last): ... TypeError: foo() may not have **kwargs If you need the functionality of **kwargs, you can request it by adding a final argument with the name _kwargs. The keyword arguments will be passed in (as a dict) through this argument:: >>> @positional_only ... def foo(a, b=None, _kwargs=None): ... return a, b, _kwargs ... >>> foo(1) (1, None, {}) >>> foo(1, 2) (1, 2, {}) >>> foo(1, bar=42, baz='spam') (1, None, {'baz': 'spam', 'bar': 42}) When your function is called with the wrong number of arguments, your callers will only see the number of arguments they expected. That is, argument counts will not include the _kwargs argument:: >>> foo() Traceback (most recent call last): ... TypeError: foo() takes at least 1 positional argument(s) (0 given) >>> foo(1, 2, 3) Traceback (most recent call last): ... TypeError: foo() takes at most 2 positional argument(s) (3 given) Note that unlike a normal **kwargs, using _kwargs in a @positional_only function causes *all* keyword arguments to be collected, even those with the same names as other function parameters:: >>> foo(1, a=42, b='spam') (1, None, {'a': 42, 'b': 'spam'}) >>> foo(1, _kwargs='this is just silly') (1, None, {'_kwargs': 'this is just silly'}) This is of course because all function parameters are considered to be positional only and therefore unnamed for the purposes of argument parsing. Note that you cannot use *args or **kwargs with the _kwargs argument:: >>> @positional_only ... def foo(a, b, _kwargs, *args): ... pass ... Traceback (most recent call last): ... TypeError: foo() may not have *args >>> @positional_only ... def foo(a, b, _kwargs, **kwargs): ... pass ... Traceback (most recent call last): ... TypeError: foo() may not have **kwargs """ import doctest import inspect import functools def positional_only(func): name = func.__name__ # don't allow **kwargs arg_names, args_name, kwargs_name, _ = inspect.getargspec(func) if kwargs_name is not None: raise TypeError('%s() may not have **kwargs' % name) # if no keyword arguments were requested, create a wrapper that # only accepts positional arguments (as *args) if not arg_names or arg_names[-1] != '_kwargs': # wrapper that raises an error for **kwargs def positional_wrapper(*args, **kwargs): if kwargs: msg = '%s() does not allow keyword arguments' raise TypeError(msg % name) return func(*args) # if keyword arguments were requested (through a final _kwargs), # create a wrapper that collects **kwargs and passes them in as # the final positional argument else: # don't allow *args if args_name is not None: msg = '%s() may not have *args' raise TypeError(msg % func.__name__) # determine defaults and expected argument counts defaults = func.func_defaults defaults = defaults and defaults[:-1] or [] n_expected = func.func_code.co_argcount - 1 min_expected = n_expected - len(defaults) def positional_wrapper(*args, **kwargs): # raise a TypeError for wrong number of arguments here # because func() needs to take the extra 'kwargs' # argument but the caller shouldn't know it arg_count_err = None if len(args) < min_expected: arg_count_err = 'at least %i' % min_expected if len(args) > n_expected: arg_count_err = 'at most %i' % n_expected if arg_count_err is not None: msg = '%s() takes %s positional argument(s) (%i given)' raise TypeError(msg % (name, arg_count_err, len(args))) # fill in defaults and add the final _kwargs argument, # then call the function with only positional arguments n_missing = n_expected - len(args) args += func.func_defaults[-n_missing - 1:-1] args += (kwargs,) return func(*args) # return the wrapped function functools.update_wrapper(positional_wrapper, func) return positional_wrapper if __name__ == '__main__': doctest.testmod() STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

2 4

dict.fromkeys() better as dict().setkeys() ? (and other suggestions)
by Ron Adam 30 May '07

30 May '07

The dictionary fromkeys method seems out of place as well as miss-named. IMHO In the current 2.5 branch it occurs only twice. (excluding tests) ---- $ grep -r "fromkeys" Lib/*.py Lib/httplib.py: header_names = dict.fromkeys([k.lower() for k in headers]) Lib/UserDict.py: def fromkeys(cls, iterable, value=None): ---- In httplib.py, it is used as a set to remove duplicates. There are enough correct uses of it in the wild to keep the behavior, but it can be done in a better way. I feel it really should be called set_keys and implemented as a method that operates on the current dictionary instead of being a constructor for a new dictionary. That will allow you to add keys with a default value to an already existing dictionary, or to create a new one with a dictionary constructor. dict().set_keys(s, v=None) # The current fromkeys behavior. I think this reads better and can be used in a wider variety of situations. It could be useful for setting an existing dictionary to a default state. # reset status of items. status.set_keys(status.keys(), v=0) Or more likely, resetting a partial sub set of the keys to some initial state. The reason I started looking at this is I wanted to split a dictionary into smaller dictionaries and my first thought was that fromkeys would do that. But of course it doesn't. What I wanted was to be able to specify the keys and get the values from the existing dictionary into the new dictionary without using a for loop to iterate over the keys. d = dict(1='a', 2='b', 3='c', 4='d', 5='e') d_odds = d.from_keys([1, 3, 5]) # new dict of items 1, 3, 5 d_evens = d.from_keys([2, 4]) # new dict of items 2, 4 There currently isn't a way to split a dictionary without iterating it's contents even if you know the keys you need before hand. A from_keys method would be the inverse complement of the update method. A del_keys method could replace the clear method. del_keys would be more useful as it could operate on a partial set of keys. d.delkeys(d.keys()) # The current clear method behavior. Some potentially *very common* uses: # This first one works now, but I included it for completeness. ;-) mergedicts(d1, d2): """ Combine two dictionaries. """ dd = dict(d1) return dd.update(d2) splitdict(d, keys): """ Split dictionary d using keys. """ keys_rest = set(d.keys()) - set(keys) return d.from_keys(keys), d.from_keys(keys_rest) split_from_dict(d, keys): """ Removes and returns a subdict of d with keys. """ dd = d.from_keys(keys) d.del_keys(keys) return dd copy_items(d1, d2, keys): """ Copy items from dictionary d1 to d2. """ d2.update(d1.from_keys(keys)) # I really like this! move_items(d1, d2, keys): """ Move items from dictionary d1 to d2. """ d2.update(d1.from_keys(keys)) d1.del_keys(keys) I think the set_keys, from_keys, and del_keys methods could add both performance and clarity benefits to python. So to summarize... 1. Replace existing fromkeys method with a set_keys method. 2. Add a partial copy items from_keys method. 3. Replace the clear method with a del_keys method. So this replaces two methods and adds one more. Overall I think the usefulness of these would be very good. I also think it will work very well with the python 3000 keys method returning an iterator. (And still be two fewer methods than we currently have.) Any thoughts? Cheers, Ron

5 18

Re: [Python-ideas] proposal to add rowexpr as a keyword
by Steve Howell 30 May '07

30 May '07

--- Steve Howell <showell30(a)yahoo.com> wrote: > > --- Josiah Carlson <jcarlson(a)uci.edu> wrote: > > It could, but I would happily bet you $100 that it > > won't. > > Maybe not by 2010, but I'll make a gentlemen's bet > that native SQL is in Python by 2005. > ...when I come back in my time machine. 2015 ____________________________________________________________________________________Sick sense of humor? Visit Yahoo! TV's Comedy with an Edge to see what's on, when. http://tv.yahoo.com/collections/222

1 0

SQL-like way to manipulate Python data structures
by Steve Howell 30 May '07

30 May '07

I find myself often writing somewhat tedious code in Python that is just slicing/aggregating fairly simple data structures like a list of dictionaries. Here is something that I'm trying to express, roughly: charges = sum float(charge) from events on setup, install group by name Here is the Python code that I wrote: def groupDictBy(lst, keyField): dct = {} for item in lst: keyValue = item[keyField] if keyValue not in dct: dct[keyValue] = [] dct[keyValue].append(item) return dct dct = groupDictBy(events, 'name') for name in dct: events = dct[name] charges = {} for bucket in ('setup', 'install'): charges[bucket] = sum( [float(event['charge']) for event in events if event['bucket'] == bucket]) Comments are welcome on improving the code itself, but I wonder if Python 3k (or 4k?) couldn't have some kind of native SQL-like ways of manipulating lists and dictionaries. I don't have a proposal myself, just wonder if others have felt this kind of pain, and maybe it will spark a Pythonic solution. ____________________________________________________________________________________Sick sense of humor? Visit Yahoo! TV's Comedy with an Edge to see what's on, when. http://tv.yahoo.com/collections/222

6 12

Re: [Python-ideas] [Python-Dev] nodef
by Collin Winter 27 May '07

27 May '07

[moving to python-ideas, where it belongs] On 5/23/07, Martin Blais <blais(a)furius.ca> wrote: > I often have the need for a generic object to use as the default value > for a function parameter, where 'None' is a valid value for the > parameter. For example: > > _sentinel = object() > > def first(iterable, default=_sentinel): > """Return the first element of the iterable, otherwise the default value (if > specified). > """ > for elem in iterable: # thx to rhettinger for optim. > return elem > if default is _sentinel: > raise StopIteration > return default > > Here, 'default' could legally accept None, so I cannot use that as the > default value, nor anything else as far as I'm concerned (I don't know > what lives in the iterable, so why should I make assumptions?). > Sometimes in the past I've create generic objects, declared a class, > e.g.: > > class NoDef: pass > def foo(default=NoDef): > ... > > and lately I've even started using the names of builtin functions (it > bothers me a little bit though, that I do that). > > I think Python needs a builtin for this very purpose. I propose > 'nodef', a unique object whose sole purpose is to serve as a default > value. It should be unique, in the same sense that 'None' is unique. I've run into this situation several times before, and I've never felt that defining my own sentinel was unduly taxing. Having a built-in would only save a single line, the initial "sentinel = object()". Collin Winter

4 3

Re: [Python-ideas] [Python-Dev] nodef
by Terry Jones 24 May '07

24 May '07

Adding nodef would be another step on the road to anarchy and nihilism. It wouldn't be long before people began using nodef as a possible value for a function, and then where would we be? Besides, using xxx = object() seems like a safer bet. Your code is unlikely to be accidentally called by someone else using your semi-private xxx object. If there were a documented official nodef object, someone would eventually call you with nodef instead of None. Terry

1 0

crazy ideas
by Aaron Brady 23 May '07

23 May '07

Annotated list callers: q = d[a,b,c] is deque simple sugar for from collections import deque q = deque( a, b, c ) might also do s = s[a,b,c] for s = set([a,b,c])

10 17

Re: [Python-ideas] Positional only arguments
by Ron Adam 23 May '07

23 May '07

Aaron Brady wrote: >> -----Original Message----- >> From: python-ideas-bounces(a)python.org [mailto:python-ideas- >> bounces(a)python.org] On Behalf Of Ron Adam >> >> def f(; a, b=2, c=3;): No positional only, no keyword only > > Do you prohibit defaults in type 1, the pos-only? No > Not necc. Does this look right? > >>>> def f( a=0; b=1 ): > ... print a, b >>>> f() > 0, 1 That's fine. 'a' is positional only and 'b' is either. Having defaults and being positional only are two different things. With the above function the following would be allowed. f() f(1) f(1, 2) f(1, b=2) f(b=2) But not: (If positional only also means no assignment with keywords at call time.) f(a=1) # Keyword not allowed for 'a' at call time. f(b=2, a=1) # 'a' is positional only > Also, does this look right? > >>>> def f( a=0; b ): > ... print a, b > File "<stdin>", line 1 > SyntaxError: non-default argument follows default argument Looks ok to me. Even the error is correct. > Any more groups? So far I have pos-only, pos-or-kw, and kw-only. > > If not, a ascii character works fine to delimit, like * and **. Any character can possibly do it, but does it make sense given that they already have a meaning? I have a feeling we are trying to do too much at once. required optional with default with out default in order in order only with keyword with keyword only Can these be ordered into *any* decision tree that makes sense? > Do we want to annotate each parameter individually? This could get out of > hand. But decorators won't let you do > > @f( k ) > def j( k ): > ... > > where > > @f( 'k' ) > def j( k ): > ... > > gets unwieldy and could show promise. Cf. Steven's positional only > arguments decorator. I think we should let Steven's decorator be developed further. It will at least give a way to test ideas and find more use cases. Then if there are some frequent and common use cases. Then look at what is actually needed. Cheers, Ron

3 3

Have list/deque grow a copy() method
by Ian D. Bollinger 22 May '07

22 May '07

It would be nice if containers had a more consistent interface. The standard idiom for copying a list has been L[:], but this isn't polymorphic and perhaps arcane. Also, deques can only be copied with the copy module's copy function (or calling the __copy__ wrapper directly); thus using copy.copy is the only polymorphic way to copy a mutable container. Additionally, having a consistent interface for mutable containers may dovetail with the ABC PEP. An alternate proposal would be to move copy.copy (and perhaps copy.deepcopy) into __builtins__. (Though this might be nice regardless.) As an aside, in addition to the copy method, list is the only mutable container not to have a clear method. Of course, del L[:] is the standard idiom for doing so, but this has the same problem of not being polymorphic. Perhaps having both del L[:] and L.clear() would violate the TOOWTDI principle, but L.reverse() can also be done with L = L[::-1] (maybe not the best example since they have different semantics.) Nevertheless, one is a lot easier to read. As for a use case, one might be a class that generalizes how it stores elements. For instance... class MutableUnorderedTree(AbstractTree): _storage_class = set class MutableOrderedTree(AbstractTree): _storage_class = list ...the latter has to be adapted to ensure consistent interface for methods like copy/clear. Well, that's my crackpot idea for today. - Ian D. Bolligner

5 11

Positional only arguments
by George Sakkis 22 May '07

22 May '07

This was posted on python-dev two weeks ago [1] but didn't get very far; probably python-ideas is a more appropriate list if this is to be taken any further. In short, PEP 3102 enables the declaration of keyword-only arguments, making it easier for an API to grow in a backwards compatible way. As Benji York, I see the utility of positional-only arguments, not just for the sake of symmetry but for pragmatic reasons; in fact the same reasons that serve as rationale to the keywords-only PEP. I don't have any concrete proposal (syntactic or otherwise) at this point, but I'd like to see this feature in Py3K even if I dislike the specific syntax or other machinery that implements it (just like decorators). For now it's worth just getting an idea of whether the feature itself is deemed useful enough or the current status quo is adequate. George [1] http://mail.python.org/pipermail/python-dev/2006-May/064790.html -- "If I have been able to see further, it was only because I stood on the shoulders of million monkeys." Unknown

15 46

Re: [Python-ideas] Positional only arguments
by Steven Bethard 21 May '07

21 May '07

On 5/21/07, Aaron Brady <castironpi(a)comcast.net> wrote: > Question on it, the example. Is this not a feature of all UserDict > subclasses? Yet we have that anyway. > > >>> from UserDict import * > >>> class C(UserDict): > ... pass > ... > >>> c= C( a=1, b=2 ) > >>> print c > {'a': 1, 'b': 2} > >>> c.update( self=3 ) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: update() got multiple values for keyword argument 'self' Yes, it's a bug still present in UserDict. Using the @positional_only decorator I posted in a separate thread, we can make it work correctly:: >>> class C(object): ... @positional_only ... def __init__(self, container=None, _kwargs=None): ... print container, _kwargs ... >>> C(self=1, container=2) None {'self': 1, 'container': 2} STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

1 0

Re: [Python-ideas] Positional only arguments
by Steven Bethard 21 May '07

21 May '07

On 5/18/07, Aaron Brady <castironpi(a)comcast.net> wrote: > Steven Bethard wrote: > > On 5/18/07, Jim Jewett <jimjjewett(a)gmail.com> wrote: > > > How could there ever be an argument that *needs* to be positional? > > > > As I've mentioned before, a great example of this is the dict() > > signature. It takes an optional "container" argument, followed by > > arbitrary keywords. If you write this in the naive way:: > > > > def __init__(self, container=None, **kwargs) > > ... > > for key, value in kwargs.items(): > > self[key] = value > > > > then you get TypeErrors for the following code:: > > > > d.update(sequence=1, container=2) > > d.update(other=1, self=2) > > > > That is, the **kwargs can never include anything named 'self' or > > 'container'. If you could declare these two arguments as > > positional-only, you'd never run into this problem. > > Might you use d.update(container=2, sequence=1)? 'Can never' caught me up. Not sure I understand the question. My point was only that you'll never reproduce this behavior:: >>> dict(self=1, other=2) {'self': 1, 'other': 2} >>> d = {} >>> d.update(container=2, sequence=1) >>> d {'container': 2, 'sequence': 1} Of course, you can rename the 'self' and 'container' parameters to something else, but that just means that you can't use whatever new names you choose. The only way to solve this is to use real positional-only arguments, which currently means using *args and parsing out things within the function body. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

6 10

Implementing PEP 3130 using decorators
by Arnaud Delobelle 20 May '07

20 May '07

Hi PEP 3130, 'Access to Current Module/Class/Function', proposed the introduction of new keywords (__class__, __function__, and __module__) to refer to what lexical unit theses names were in. It was rejected, but I have thought about a way to produce the same functionality without the need of new keywords, by using decorators instead. It would go like this: class Foo(object): # some defs... @bindclass def bar(thisclass, self): # Within this method the name 'thiclass' refers to class Foo # ... @bindclass def bar(Foo, self): # The name of 'thisclass' can be chosen, just like 'self' can. # Within this method the name 'Foo' refers to class Foo, even # after Foo is bound to another object # ... # More defs... @bindfunction def factorial(thisfunction, n): # Within this function the name 'thisfunction' refers to the factorial # function (with only one argument), even after 'factorial' is bound # to another object if n > 0: return n * thisfunction(n - 1) else: return 1 Here is an implementation: class bindclass(object): def __init__(self, f): self.f = f def bind(self, cls, attr): def bound_m(*args, **kwargs): return self.f(cls, *args, **kwargs) bound_m.__name__ = attr self.m = bound_m def __get__(self, obj, objtype=None): return self.m.__get__(obj, objtype) class Type(type): def __init__(self, name, bases, attrs): for attr, val in attrs.iteritems(): if isinstance(val, bindclass): val.bind(self, attr) class Object(object): __metaclass__ = Type def bindfunction(f): def bound_f(*args, **kwargs): return f(bound_f, *args, **kwargs) bound_f.__name__ = f.__name__ return bound_f And an example: class Foo(Object): @bindclass def foo(this_class, self): return this_class, self class Bar(Foo): @bindclass def bar(this_class, self): return this_class, self f = Foo() b = Bar() f.foo() # -> Foo, Foo object b.foo() # -> Foo, Foo object f.bar() # -> Bar, Foo object -- Arnaud

1 0

Re: [Python-ideas] Positional only arguments
by Arnaud Delobelle 20 May '07

20 May '07

On Wed, May 16, 2007 5:25 pm, George Sakkis wrote: [...] > > What's the future of "private" attributes in Py3K ? I remember Raymond > Hettinger bringing up a challenge on c.l.py about a decorator that > could replace the double underscore mangling and Michele Simionato > caming up with a bytecode hack that showed it is possible. IIRC Michele Simionato made a 'CurrentClass' name available to methods, which contains the class that the method is defined in. While interesting, this doesn't help create 'private' attribute, as name mangling does. In fact it is possible to implement this in pure Python using decorators and metaclasses. I tried to take up the challenge (without bytecode hackery, which I am not capable of) but it turned out you need to chase around the call stack a lot! -- Arnaud

1 0

Re: [Python-ideas] Positional only arguments
by George Sakkis 18 May '07

18 May '07

On 5/16/07, Benji York <benji(a)zope.com> wrote: > Guido van Rossum wrote: > > Personally, when I really wanted this, I've given the arguments names > > like __foo. > > That could be enshrined as the syntax, much like "private" attributes. What's the future of "private" attributes in Py3K ? I remember Raymond Hettinger bringing up a challenge on c.l.py about a decorator that could replace the double underscore mangling and Michele Simionato caming up with a bytecode hack that showed it is possible. If they're staying though, +epsilon for blessing __args as positional-only (for some positive epsilon ;-)). George -- "If I have been able to see further, it was only because I stood on the shoulders of million monkeys."

5 6

dot bend spoon
by Aaron Brady 17 May '07

17 May '07

Is the going semantic to carry a relation: spoon.bend() or johnson.bend( spoon ) ? How to phrase this. How do -you- do it? Weird. Because spoon.bend() is a misnomer; construed to mean spoon.getsbent(). More in the predicate vs. controller category. Haven't got model-view-controller yet but I know the term.

4 3

Re: [Python-ideas] parameter omit
by Ron Adam 15 May '07

15 May '07

Aaron Brady wrote: >> -----Original Message----- >> From: Aaron Brady [mailto:castironpi@comcast.net] >> Sent: Sunday, May 13, 2007 9:28 PM >> >>> -----Original Message----- >>> From: python-ideas-bounces(a)python.org [mailto:python-ideas- >>> bounces(a)python.org] On Behalf Of Aaron Brady >>> >>> class GuardDefault: >>> Val= object() >>> def __call__( self, *args, **kwargs ): >>> args=list(args) >>> for i,j in enumerate( args ): >>> if j is GuardDefault.Val: >>> offset= >>> self.callable.func_code.co_argcount-\ >>> len(self.callable.func_defaults) >>> args[i]= >>> self.callable.func_defaults[i-offset] >>> return self.callable( *args,**kwargs ) >>> def __init__( self,callable ): >>> self.callable= callable >> Do you have any interest in pursuing this? > > Actually not too bad for functools. Does this take a proposal? -I- am > God-awful at lengthy technical writing. I suggest you post it on the python-list firsts and see what others think. They may offer alternative ways of doing it that will make it better based on how they will used it. Cheers, Ron

1 0

Re: [Python-ideas] parameter omit
by Aaron Brady 15 May '07

15 May '07

> -----Original Message----- > From: Aaron Brady [mailto:castironpi@comcast.net] > Sent: Sunday, May 13, 2007 9:28 PM > > > -----Original Message----- > > From: python-ideas-bounces(a)python.org [mailto:python-ideas- > > bounces(a)python.org] On Behalf Of Aaron Brady > > > > class GuardDefault: > > Val= object() > > def __call__( self, *args, **kwargs ): > > args=list(args) > > for i,j in enumerate( args ): > > if j is GuardDefault.Val: > > offset= > > self.callable.func_code.co_argcount-\ > > len(self.callable.func_defaults) > > args[i]= > > self.callable.func_defaults[i-offset] > > return self.callable( *args,**kwargs ) > > def __init__( self,callable ): > > self.callable= callable > > Do you have any interest in pursuing this? Actually not too bad for functools. Does this take a proposal? -I- am God-awful at lengthy technical writing.

2 1

Function/method call issues
by Raj B 14 May '07

14 May '07

Hello everyone, First of all, I'll introduce myself. I've been working on a Python compiler written in OCaml for about a year now. Some of you might know me from my talk at Google. One of the list members suggested this mailing list as a good forum for discussion and bouncing ideas. I think that one issue to consider are the complex semantics of functions and method calls. Whenever a function call is made, there is a lot of checking involved as to 'where' the function is called from: i.e. whether it's a user-defined function/method, a built-in function/method, a class function/method or an instance function/ method. Just reading the reference manual section on 'function objects' is enough to see how many different cases are there. Of course, this provides tons of flexibility in reusing the same code for different ways. However, it increases the amount of book-keeping and pre-processing required for a call. For example, Consider an object that implements methods 'str' and 'append' (the list object, for instance). During retrieval of methods, the following default retrieval method is called (in OCaml syntax, a simplified version). let list_getattr o attr = let v = get o.dict attr in (* lookup the attribute in the dictionary *) match attr with | String "append" | ... | -> v.instance = o; v (* set the method's instance to o before returning it *) | _ -> v (* else return the method object as is *) So basically, I have to check for two cases based on the actual names of the methods being called. This seems like extra work for the runtime. As a suggestion, I think that adopting a single calling convention (either 'x.foo()' or 'foo(x)') would have 2 advantages. 1) It would make it easier for users and make for more readable code. I still find myself having to refer to the docs to find out how a method needs to be called. 2) It would improve performance by reducing these checks shown above to one case. What do y'all think? Would this sacrifice too much flexibility? If this topic has been discussed on this list before, please feel free to point me there. Thank you Raj

3 4

python appreciation site
by Aaron Brady 14 May '07

14 May '07

This is a joke, for austerity. http://home.comcast.net/~castironpi/pythonappr.html

2 2

parameter omit
by Aaron Brady 14 May '07

14 May '07

Is it possible to signal to a parameter to use its default? def f( a, b=None, c='' ):... f( 123, <def>, 'abc' ) f( 123, ObjA, <def> ) Did I miss something, plus did you cover it before?

5 9

Re: [Python-ideas] parameter omit
by Tal Einat 13 May '07

13 May '07

On 5/11/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > > Perhaps we could learn from experience? > > > > I've recently had to automate Excel with Python and used the COM > > interface. In COM, there is a value called "Missing" which is passed to > > functions when no value should be passed for an argument - similar to > your > > DefaultArg. This leads to code like this (C# code examples): > > > > http://msdn2.microsoft.com/en-us/library/aa168343(office.11).aspx > > (straight from the online MSDN) > > > > MenuBarItem = MainMenuBar.Controls.Add( > > Office.MsoControlType.msoControlPopup, > > Type.Missing, Type.Missing, Type.Missing, true); > > > > And that was not a far-fetched example - here is an extreme example: > > > > http://blogs.ittoolbox.com/visualbasic/evolution/archives/importing- > > unformatted-excel-sheets-into-sql-server-2000-using-net-part-i-7501 > > <http://blogs.ittoolbox.com/visualbasic/evolution/archives/importing- > > unformatted-excel-sheets-into-sql-server-2000-using-net-part-i-7501> > > > > Excel.Application excelApp = new Excel.ApplicationClass(); > > Excel.Workbook thisWorkbook = > > excelApp.Workbooks.Open(strFileName,Type.Missing, Type.Missing, > > Type.Missing, Type.Missing,Type.Missing, Type.Missing, Type.Missing, > > Type.Missing,Type.Missing, Type.Missing, Type.Missing, > > Type.Missing,Type.Missing, Type.Missing); > > > > > > IMO this is horrendous. Horrible. Eye-gouging. > > > > I'd rather write machine code in hex. > > > > > > - Tal > > Ok, this isn't working to get my idea over. There's something wrong with > somebody, and it's simplest to assume its me. Probably inexperience in > posting. Not to mention, I have no idea who I'm talking to, or what the > local dialect is. (Sorry about the blank message too... wrong keystroke.) > > Tal's examples don't contradict me. In his, I call: > > > MenuBarItem = MainMenuBar.Controls.Add( > > Office.MsoControlType.msoControlPopup, thenamedarg= true); > and > > Excel.Workbook thisWorkbook = excelApp.Workbooks.Open(strFileName) > > Traditional function call. Calls still will work the way they do today. > You have -extra- flexibility, not less, sir. I add zero requirements. > Please let me repeat. Nothing breaks. These are simply the examples I could find with several minutes of Googling, admittedly not great examples, as you pointed out. I was just trying to show real world ugly code which was written as a result of an infrastructure adopting a concept which is very similar to the one which you propose. The point was to show that adopting your suggestion could result in even less readable function calls. Yes, nothing breaks. I really get it. I'm against this because it is, IMHO, not Pythonic. I'm not saying it won't be possible to write readable code. I'm saying it -would- be possible to write even less readable code. Specifically, I might be forced to write such ugly function calls when calling functions written by others, therefore adopting this feature could lead to my being forced to write uglier code. Not to mention reading others' uglier code... - Tal (P.S. in the first example, a keyword argument could not be used, because all of the arguments were positional.)

2 1

Re: [Python-ideas] parameter omit
by Josiah Carlson 13 May '07

13 May '07

"Aaron Brady" <castironpi(a)comcast.net> wrote: > > From: Josiah Carlson [mailto:jcarlson@uci.edu] > > "Aaron Brady" <castironpi(a)comcast.net> wrote: > > > What is the default parameter for the function? Problem is, the docs > > don't > > > say what it is. Does it pick its own filename to use for empty inputs? > > > > >>> import urllib > > >>> help(urllib.urlretrieve) > > Help on function urlretrieve in module urllib: > > > > urlretrieve(url, filename=None, reporthook=None, data=None) > > > > >>> > > No nitpicking, Mr. Carlson. These pieces of information are -not- always > available. Should I send you my 486 laptop so that you can have access to a Python interpreter? Because I really can't see how you being ignorant of the arguments to a standard library function in any way supports your case. Further, if you always use 'None' as the default for functions being called, then you can get things like this... class WebRetrieveElement: hook = None filename = None def netretrieveNew( self ): filename= self.filename or ParamDefault hook= hasattr( self,'hook' ) and self.hook or ParamDefault urlretrieve( self.url, filename, hook ) def netretrieveBest(self): urlretrieve(self.url, self.filename, self.hook) And look at that. I define two class level variables and all of a sudden the implementation of netretrieveBest gets a trivial implementation (though it breaks netretrieveNew, which uses a generally frowned upon mechanism that fails when filename and hook are None). But no, lets add a completely useless instance of object so that people can pass silly default arguments like ParamDefault that is a bazillion times more confusing than what Python users have been using for 15 years; None. - Josiah

7 17

Re: [Python-ideas] parser in stdlib
by Adam Atlas 13 May '07

13 May '07

On 10 May 2007, at 18.39, Aaron Brady wrote: > There is, but I want access to it. >>>> import parser >>>> parser.__file__ > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > AttributeError: 'module' object has no attribute '__file__' That's because it's a builtin module, written in C. It's Python/ parsermodule.c in the source distributions. What exactly are you suggesting should be possible? Are you trying to programmatically change the parser?

4 7

Alternate quote delimiters
by Ron Adam 13 May '07

13 May '07

While watching the various issues about raw strings and quotes being discussed, I decided look into other ways to resolve string delimiter collisions that might also improve pythons string handling in general. And I'd figure I'd let it get shot full of holes here first. ;-) Here's a nice overview of the issue with various examples. http://en.wikipedia.org/wiki/String_literal *Note: The wiki python raw string example has a trailing backslash which would cause an error. r"The Windows path is C:\Foo\Bar\Baz\" In the case of raw strings, the escaped quotes are only needed in order for tokenize.c to locate the end of the string. The back slashes remain in the string. The end of the string is the first unescaped quote of the same type as the starting quote. >>> s = r"\"Hello World\"" >>> print s \"Hello World\" This allows you to enter, back-slash + quote, pairs into a string, but ... - This doesn't help in entering only quotes. You still need to either use a different quotes than what is in the string or not use raw strings. - A minor side effect is that you can't have a raw string end with a single slash as in the incorrect wiki example above. Other languages which use escape characters in raw strings have the same issue. - If you have a long string with multiple quotes as you can in regular expressions the increased number of \ characters can make the regular expression more difficult to read and can add up to be more characters than needed. - Another minor issue I've found is doctest that do test on quoted strings may have multiple nested doc strings which may require careful selection of quote characters. The problem arises because the start and end delimiters are the same. The start of the nested string ends the top level string. The current solution is to pick different quote characters or escape quote characters in the nested strings. So how about a multiple quoting solution? The examples of this in the wiki page are of the form... qq^I said, "Can you hear me?"^ qq@I said, "Can you hear me?"@ qq§I said, "Can you hear me?"§ But that doesn't fit well with pythons use of quotes. But a variation of this would. Add a 'q' string prefix similar to the 'u' and 'r' prefix's that takes the first character inside the quotes as an additional delimiter. Then ending quote will then need to have that same character proceeding it. q"^I said, "Can you hear me?"^" q"""|I said, "Can you hear me?"|""" The vertical bar is part of the quote here, not part of the string. rq"^The Windows path is C:\Foo\Bar\Baz\^" This example will work as expected. Because the beginning and ending of the strings are not identical, it's possible that they can allow nesting. rq"""^ rq"""^ This is a nested string. ^""" """ Another nested string. """ ^""" The most useful feature of this would be in temporarily commenting out large blocks of python code. Currently this doesn't work well if the block that contain triple quoted doc strings. Another option might be to designate certain quote delimiters for special purposes. Dedented strings. q"""< This is a Dedented Paragraph. <""" Commented out source code. q"""# def foo(bar): """ A bar checker. """ return bar is True #""" ReST formatted strings. rq""": Bullet lists: - This is item 1 - This is item 2 - Bullets are "-", "*" or "+". Continuing text must be aligned after the bullet and whitespace. Note that a blank line is required before the first item and after the last, but is optional between items. :""" This use would require some way to preserve it's quoting type so it can later be used to render the text. (Any ideas?) Cheers, Ron

5 6

Re: [Python-ideas] parser in stdlib
by Aaron Brady 13 May '07

13 May '07

> -----Original Message----- > From: george.sakkis(a)gmail.com [mailto:george.sakkis@gmail.com] On Behalf > Of George Sakkis > Sent: Thursday, May 10, 2007 10:36 PM > To: Aaron Brady > Subject: Re: [Python-ideas] parser in stdlib > > On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > Huge bag of worms, I see now. I was tinkering for hobby Python use. I > > hadn't proposed a syntax change, not there yet. I was wanting to > intercept > > parser somewhere after it's started parsing source, but before it gets > to > > the rules. The particular change I'm tinkering with was replacing an > equal > > sign with a natural word. > > > > 1, 2 to a, c > > -and- > > to a, c 1, 2 > > > > map to: > > > > a, c = 1, 2 > > A perfect example of why programmable syntax is out of question for > Python. > > George Hence the quote. First thing I said was, "...it might not be advisable either, subject to abuse, per GvR...." But that doesn't preclude exposing `parser'. It is the extent of my question, no more. Can we expose the module? Do I take the powers that be to have said, "No, that would open too many doors and give the programmers too much freedom?" If so, that's straight-forward and honest, and presents the next problem for solutions. Can we expose that and still keep programs up to standard?

7 10

Re: [Python-ideas] the future of the GIL
by Aahz 12 May '07

12 May '07

[excessive quoting ahead to move to python-ideas from python-3000, please trim when you followup] On Wed, May 09, 2007, Talin wrote: > Greg Ewing wrote: >> Giovanni Bajo wrote: >>> >>> using multiple processes cause some >>> headaches with frozen distributions (PyInstaller, py2exe, etc.), >>> like those usually found on Windows, specifically because Windows >>> does not have fork(). >> >> Isn't that just a problem with Windows generally? I don't >> see what the method of packaging has to do with it. >> >> Also, I've seen it suggested that there may actually be >> a way of doing something equivalent to a fork in Windows, >> even though it doesn't have a fork() system call as such. >> Does anyone know more about this? > > I also wonder about embedded systems and game consoles. I don't know how > many embedded microprocessors support fork(), but I know that modern > consoles such as PS/3 and Xbox do not, since they have no support for > virtual memory at all. > > Also remember that the PS/3 is supposed to be one of the poster children > for multiprocessing -- the whole 'cell processor' thing. You can't write > an efficient game on the PS/3 unless it uses multiple processors. > > Admittedly, not many current console-based games use Python, but that > need not always be the case in the future, and a number of PC-based > games are using it already. > > This much I agree: There's no point in talking about supporting multiple > processors using threads as long as we're living in a refcounting world. > > Thought experiment: Suppose you were writing and brand-new dynamic > language today, designed to work efficiently on multi-processor systems. > Forget all of Python's legacy implementation details such as GILs and > refcounts and such. What would it look like, and how well would it > perform? (And I don't mean purely functional languages a la Erlang.) > > For example, in a language that is based on continuations at a very deep > level, there need not be any "global interpreter" at all. Each separate > flow of execution is merely a pointer to a call frame, the evaluation of > which produces a pointer to another call frame (or perhaps the same > one). Yes, there would still be some shared state that would have to be > managed, but I wouldn't think that the performance penalty of managing > that would be horrible. One of the primary goals of Python was/is to be an easy glue language for C libraries. How do you propose to handle the issue that many C libraries use global state? -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet

1 0

Re: [Python-ideas] parameter omit
by Jan Kanis 12 May '07

12 May '07

On Fri, 11 May 2007 20:09:39 +0200, Aaron Brady <castironpi(a)comcast.net> wrote: >> -----Original Message----- >> From: Jan Kanis [mailto:jan.kanis@phil.uu.nl] >> Sent: Friday, May 11, 2007 6:59 AM >> >> In short, any new addition has to have a *lot* of utility to be >> acceptable, or else it wil lead to the way of the Perl side. > > Agreed entirely. You mean, like this? > > Who brings up the manual? Good. Document something. It isn't really about the manual, and I think I meant something like 'language reference' (sorry, not native English). It's about the complexity of the language. The size of the language reference can be seen as a more or less objective way to measure language complexity. Your proposal adds a little bit more complexity, and most people here agree that the benefits, which are there, are not worth it.

1 0

Re: [Python-ideas] parameter omit
by Steven Bethard 11 May '07

11 May '07

Steven Bethard [mailto:steven.bethard@gmail.com] > Can you show me code that would make your new "morpheme" work? All > the implementations I can imagine involve making it some sort of > special keyword. On 5/11/07, Aaron Brady <castironpi(a)comcast.net> wrote: > Nope, routine object. I defined it earlier: > > paramdefault= object() Yes, but how is it going to *work*? Say I wrote the function:: def foo(bar=1, baz=2): print bar, baz Now if I call that like:: foo(bar=paramdefault, baz=paramdefault) I'm going to see something like:: <object object at 0x009A0468> <object object at 0x009A0468> Is that really what you want? I thought you wanted this to work for all functions... STeve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

3 2

Re: [Python-ideas] parameter omit
by Steven Bethard 11 May '07

11 May '07

On 5/11/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > -----Original Message----- > > From: Steven Bethard [mailto:steven.bethard@gmail.com] > > Sent: Friday, May 11, 2007 1:50 AM > > > > On 5/11/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > Take a shot at this one. > > > > > > class WebRetrieveElement: > > > def netretrieveOld( self ): > > > if hasattr( self,'hook' ) and self.filename: > > > urlretrieve( self.url, self.filename, self.hook > > ) > > > elif hashook: > > > urlretrieve( self.url, reporthook= self.hook ) > > > else: > > > urlretrieve( self.url, self.filename ) > > > > def netretrieve(self): > > kwargs = {} > > if self.filename: > > kwargs['filename'] = self.filename > > if hasattr(self, 'hook'): > > kwargs['reporthook'] = self.hook > > urlretrieve(self.url, **kwargs) > > Right. That's the alternative, -as- I keep bringing up. > > Now let's compare, and judge if X is worth Y. > > My solution, 3 lines. Yours, 6. If you really care about lines and characters, you can write:: k = {} if self.filename: k['filename'] = self.filename if hasattr(self, 'hook'): k['reporthook'] = self.hook urlretrieve(self.url, **k) That's 4 lines, 132 characters, compared to your 3 lines, 133 characters. But the fact that you're drawing the "lets count lines of code" card suggests that there's no point discussing this any further. Good luck with your proposal. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

2 1

Re: [Python-ideas] parameter omit
by Steven Bethard 11 May '07

11 May '07

On 5/11/07, Aaron Brady <castironpi(a)comcast.net> wrote: > Take a shot at this one. > > class WebRetrieveElement: > def netretrieveOld( self ): > if hasattr( self,'hook' ) and self.filename: > urlretrieve( self.url, self.filename, self.hook ) > elif hashook: > urlretrieve( self.url, reporthook= self.hook ) > else: > urlretrieve( self.url, self.filename ) def netretrieve(self): kwargs = {} if self.filename: kwargs['filename'] = self.filename if hasattr(self, 'hook'): kwargs['reporthook'] = self.hook urlretrieve(self.url, **kwargs) STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

2 1

Re: [Python-ideas] parameter omit
by Josiah Carlson 11 May '07

11 May '07

Adding python-ideas in as a CC, because I'm a jerk. "Aaron Brady" <castironpi(a)comcast.net> wrote: > On a personal note, Josiah, you strike first. "Pestering" is pejorative. > My statements do not merit it. Keep it off the newsgroup. You insult me > without grounds; and in fact -after- my abilities are demonstrated. Quote > me instead, as I do, to disagree. > > The discussion reads as follows. "Better except that it has no support." > So support it. Who's to blame: myself, or those with pull? > > I have a good idea by newsgroup standards. You reveal bias to summarily > dismiss. I urge you to reconsider; practice diplomacy here. Post what you want. If people aren't offering +1 or -1, it could be for any number of reasons; from being busy (it is in the middle of the week), to just not caring enough either way, believing that their opinions and interests are already being expressed, etc. But when you post something, then 24 hours and 40 minutes later you ask, "Also, any follow-up on this? (I posted at top.)" at the bottom of a 95 line quote of your own last post without cutting out the unimportant parts, it is pestering, and a very low signal to noise ratio 1:95. Never mind that you only recently seemed to learn that we don't top post in Python mailing lists, but you still haven't realized that we trim unimportant parts of quoted posts. Had you top posted or failed to trim either in python-list (or the equivalent comp.lang.python), any number of people would have told you as much. But hey, you are being such the good usenet poster, you couldn't have made a mistake, of course not. Get over yourself. With that said, I don't particularly like the addition of syntax for something that can already be done other ways (TOOWTDI), or the addition of a new builtin to signal something that I've never heard *anyone* ask about before. Really, the only *reasonable* name is 'default', which is used at least 146 times in the Python standard library (according to my regular expression of default\s*=), and I'm sure thousands of times in 3rd party code. It would necessitate a huge change for a feature that no one (but you) as asked for so that you can use... foo(a, b=default, c=1234) foo(a, b=253463, c=default) rather than... foo(a, c=1234) foo(a, b=253463) Personally, I find the latter more consistant (only pass arguments that you care to change). If you don't, that's fine. The burdon of proof isn't on me, or anyone else in python-ideas to prove to you that your idea sucks; it's on you to prove to everyone else that your idea *doesn't* suck. Good luck on that. Local jerk, - Josiah

3 6

Re: [Python-ideas] parameter omit
by Talin 11 May '07

11 May '07

Aaron Brady wrote: >>> Specific and concise. Clearly better, by all measures I read. >> With PEP 3102 in place you don't even need the **kwargs: >> >> http://www.python.org/dev/peps/pep-3102/ >> >> -- Talin > > I'm not adding restrictions. I'm simplifying. The subject solution is > general. Less code in all cases, meaning still as clear or clearer. Non-sequitur? How does that relate to what I said? Let me state in different words what others have already stated: There are already ways to accomplish most what you are trying to do. In Python, any argument will have its default value if you omit it from the argument list when calling the function. There's no need to explicitly tell it to use the default value, it just does this naturally. Now, the only case where this might be a problem is a fairly rare and odd case where you are specifying arguments positionally and not by keyword, where you want to specify the value of argument N+1 but want argument N to be its default. But the language should not make a special cases to support this particular case, for 3 reasons: 1) The case is rare. 2) There are easy ways around it: Simply supply the value of argument N+1 by keyword instead of by position. 3) If your function has lots of positional arguments, each of which has a different meaning, then your function probably needs to be refactored anyway. Using keywords is much safer in such cases, and if you use keywords, then the whole problem you raise goes away. -- Talin

2 1

Re: [Python-ideas] parameter omit
by Josiah Carlson 11 May '07

11 May '07

"Aaron Brady" <castironpi(a)comcast.net> wrote: > Also, any follow-up on this? (I posted at top.) I don't like it. The current calling semantics are sufficient for the vast majority of cases. For those cases that are not covered by the current calling semantics, there is a PEP for allowing variations in optional arguments, keyword arguments, etc. I can't remember the number, but the PEP index has it. As for signaling "use the default", there is a standard method: omit the argument. If you want the argument to always be required to be a keyword argument, you can use... def foo(arg1, **kwargs): arg2 = kwargs.get('arg2', 1.2325) arg3 = kwargs.get('arg3', 'hello') ... - Josiah

3 3

Re: [Python-ideas] parameter omit
by Steven Bethard 11 May '07

11 May '07

On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > From: Steven Bethard [mailto:steven.bethard@gmail.com] > > On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > The new way: > > > > > > my_b, my_c= paramdefault, paramdefault > > > [code to assign my_b and my_c] > > > foo(a, my_b, my_c) [snip] > > It is so extremely uncommon to need to do such a thing that I'm firmly > > against adding any additional syntax. Syntax should only be introduced > > for common things that many people need. > > Mr. Bethard, I am not adding syntax. Don't put words in my mouth. This is > another one- observe the subject line. Tell me you always make a call with > the same parameters, every time you make the call. My thinking can't be > that backwards. A use case could take some time, but sure. Hmm... If ``paramdefault`` isn't syntax, then how are you proposing to implement it? STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

1 0

Re: [Python-ideas] parameter omit
by Josiah Carlson 11 May '07

11 May '07

"Aaron Brady" <castironpi(a)comcast.net> wrote: > To finish up, I, the newcomer, have a solution that encompasses more > situations at very small cost. No compatibility issues either. What should > I expect? Is this the same idea or something different? If it's the same, wait a few days for responses. If it is something different, post the current situation, what's wrong with the current situation, and what you think would be better. When people disagree with you (there is always at least one :) ), don't reply with "but that doesn't do *exactly* what I want". Instead understand that no matter how much you post, no matter how much people like your idea, no matter how much Guido likes your idea, or even if you have a working implementation, it can be tossed in the trash (see the February "New syntax for 'dynamic' attribute access" postings here and the related python-dev thread for an example). - Josiah

1 0

Re: [Python-ideas] parameter omit
by Josiah Carlson 11 May '07

11 May '07

"Aaron Brady" <castironpi(a)comcast.net> wrote: > Library functions have many parameters, and huge if statements are hard to > read. My solution costs only a single built-in object, not even a keyword > or syntax modification. Your solution takes a three-line function > definition. Compare to mine: > > def foo(arg1, argFoo=1.2325, argBree='hello'): > ... > > Specific and concise. Clearly better, by all measures I read. Except that it has no support from anyone with any pull. Feel free to wait for it, and/or keep pestering people to offer their opinions. Note that not everyone has the desire for such (arguably unnecessary) consistency as you seem to. - Josiah

2 2

Re: [Python-ideas] parser in stdlib
by Terry Jones 11 May '07

11 May '07

>>>>> "Josiah" == Josiah Carlson <jcarlson(a)uci.edu> writes: Josiah> "Aaron Brady" <castironpi(a)comcast.net> wrote: >> Huge bag of worms, I see now. I was tinkering for hobby Python use. I >> hadn't proposed a syntax change, not there yet. I was wanting to intercept >> parser somewhere after it's started parsing source, but before it gets to >> the rules. The particular change I'm tinkering with was replacing an equal >> sign with a natural word. Josiah> Use Logix: http://www.livelogix.net/logix/ Use sed: http://www.gnu.org/software/sed/ :-) Terry

1 0

Re: [Python-ideas] parser in stdlib
by Collin Winter 11 May '07

11 May '07

On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > -----Original Message----- > > From: Collin Winter [mailto:collinw@gmail.com] > > Sent: Thursday, May 10, 2007 5:57 PM > > > > On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > > -----Original Message----- > > > > From: python-ideas-bounces(a)python.org [mailto:python-ideas- > > > > bounces(a)python.org] On Behalf Of Adam Atlas > > > > Sent: Thursday, May 10, 2007 5:47 PM > > > > > > > > On 10 May 2007, at 18.39, Aaron Brady wrote: > > > > > There is, but I want access to it. > > > > >>>> import parser > > > > >>>> parser.__file__ > > > > > Traceback (most recent call last): > > > > > File "<stdin>", line 1, in <module> > > > > > AttributeError: 'module' object has no attribute '__file__' > > > > > > > > That's because it's a builtin module, written in C. It's Python/ > > > > parsermodule.c in the source distributions. > > > > > > > > What exactly are you suggesting should be possible? Are you trying to > > > > programmatically change the parser? > > > > > > Yes. The relevant code snippet: > > > > > > class Transformer: > > > <snip> > > > def parsesuite(self, text): > > > """Return a modified parse tree for the given suite text.""" > > > return self.transform(parser.suite(text)) > > > > > > Hence to allusion to Van Rossum's note, for the record. > > > > That is not and will not be possible using Python's built-in parser. > > > > Collin Winter > > I'm 25, FTR, heavy experience. I have no idea why that's relevant. > Why not? PEP or FAQ if there is one. Because programmable syntax isn't something we want in Python; you yourself quoted an email from Guido explaining why. If you like the idea of allowing every module to define its own syntactic constructs -- at runtime or otherwise -- Perl 6, Haskell and others are more than happy to help you. Collin Winter

3 2

Re: [Python-ideas] parameter omit
by Aaron Brady 10 May '07

10 May '07

> -----Original Message----- > From: Aaron Brady [mailto:castironpi@comcast.net] > Sent: Wednesday, May 09, 2007 5:35 PM > > Could easily incorporate, as shown here. > #sample function with default args > def f( a,b=None,c='abc' ): > print a,b,c > > #customizer > default= object() > def call_wrapper( callable, *args, **kwargs ): > args=list(args) > for i,j in enumerate( args ): > if j is default: > offset= callable.func_code.co_argcount-\ > len(callable.func_defaults) > args[i]= callable.func_defaults[i-offset] > return callable( *args,**kwargs ) > > #the uniform calls > call_wrapper( f,0,default,'def' ) > call_wrapper( f,0,'somebody',default ) > > -----Original Message----- > From: Aaron Brady [mailto:castironpi@comcast.net] > Sent: Wednesday, May 09, 2007 4:56 PM > To: 'Aaron Brady'; 'Josiah Carlson'; 'python-ideas(a)python.org' > Subject: RE: [Python-ideas] parameter omit > > You can almost do, > b=f.func_defaults[1], > but you still have to know where the defaults start. Very small what I'm > missing. > > -----Original Message----- > From: python-ideas-bounces(a)python.org > [mailto:python-ideas-bounces@python.org] On Behalf Of Aaron Brady > Sent: Wednesday, May 09, 2007 4:50 PM > To: 'Josiah Carlson'; python-ideas(a)python.org > Subject: Re: [Python-ideas] parameter omit > > No, still not uniform. Cases might be rare, even syntactic sugar maybe. > if something huge: > b=<def> > more huge > if something else: > b=mything > still more > f(a,b,c). > > -----Original Message----- > From: Josiah Carlson [mailto:jcarlson@uci.edu] > Sent: Wednesday, May 09, 2007 4:51 PM > To: Aaron Brady; python-ideas(a)python.org > Subject: Re: [Python-ideas] parameter omit > > > "Aaron Brady" <castironpi(a)comcast.net> wrote: > > > > Actually, I wanted a uniform way to call f. f(a,b,c) for both cases if > b > > can equal <def>. > > f(a, b=...) #default c > f(a, c=...) #default b > > > - Josiah > > > -----Original Message----- > > From: Steven Bethard [mailto:steven.bethard@gmail.com] > > Sent: Wednesday, May 09, 2007 3:45 PM > > To: Aaron Brady > > Cc: python-ideas(a)python.org > > Subject: Re: [Python-ideas] parameter omit > > > > On 5/9/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > > Is it possible to signal to a parameter to use its default? > > > > > > def f( a, b=None, c='' ):... > > > > > > f( 123, <def>, 'abc' ) > > > f( 123, ObjA, <def> ) > > > > In this case, it's pretty easy:: > > > > f(123, c='abc') > > f(123, ObjA) > > > > STeVe > > -- > > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > > tiny blip on the distant coast of sanity. > > --- Bucky Katt, Get Fuzzy Also, any follow-up on this? (I posted at top.)

1 0

parser in stdlib
by Aaron Brady 10 May '07

10 May '07

Is it possible to get `profile' into the stdlib? Currently just tinkering, and it might not be advisable either, subject to abuse, per GvR: "Programmable syntax is not in Python's future -- or at least it's not for Python 3000. The problem IMO is that everybody will abuse it to define their own language. And the problem with that is that it will fracture the Python community because nobody can read each other's code any more." - http://mail.python.org/pipermail/python-3000/2006-April/000286.html What do you think? Same reason apply?

3 4

Re: [Python-ideas] parser in stdlib
by Aaron Brady 10 May '07

10 May '07

> -----Original Message----- > From: Steven Bethard [mailto:steven.bethard@gmail.com] > Sent: Thursday, May 10, 2007 5:32 PM > > On 10 May 2007, at 17.34, Aaron Brady wrote: > > Is it possible to get `profile' into the stdlib? > > From: Adam Atlas [mailto:adam@atlas.st] > > What are you referring to here as `profile`? You're obviously (I > > think?) not talking about the existing stdlib module called `profile`.= > > On 5/10/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > Right. Pretend I said `parser' instead. > > I'm afraid that's not any clearer. There's also already a parser module: > > >>> import parser > >>> parser.__doc__ > "This is an interface to Python's internal parser." > > STeVe > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > There is, but I want access to it. >>> import parser >>> parser.__file__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute '__file__'

1 0

Re: [Python-ideas] parameter omit
by Aaron Brady 09 May '07

09 May '07

Could easily incorporate, as shown here. #sample function with default args def f( a,b=None,c='abc' ): print a,b,c #customizer default= object() def call_wrapper( callable, *args, **kwargs ): args=list(args) for i,j in enumerate( args ): if j is default: offset= callable.func_code.co_argcount-\ len(callable.func_defaults) args[i]= callable.func_defaults[i-offset] return callable( *args,**kwargs ) #the uniform calls call_wrapper( f,0,default,'def' ) call_wrapper( f,0,'somebody',default ) -----Original Message----- From: Aaron Brady [mailto:castironpi@comcast.net] Sent: Wednesday, May 09, 2007 4:56 PM To: 'Aaron Brady'; 'Josiah Carlson'; 'python-ideas(a)python.org' Subject: RE: [Python-ideas] parameter omit You can almost do, b=f.func_defaults[1], but you still have to know where the defaults start. Very small what I'm missing. -----Original Message----- From: python-ideas-bounces(a)python.org [mailto:python-ideas-bounces@python.org] On Behalf Of Aaron Brady Sent: Wednesday, May 09, 2007 4:50 PM To: 'Josiah Carlson'; python-ideas(a)python.org Subject: Re: [Python-ideas] parameter omit No, still not uniform. Cases might be rare, even syntactic sugar maybe. if something huge: b=<def> more huge if something else: b=mything still more f(a,b,c). -----Original Message----- From: Josiah Carlson [mailto:jcarlson@uci.edu] Sent: Wednesday, May 09, 2007 4:51 PM To: Aaron Brady; python-ideas(a)python.org Subject: Re: [Python-ideas] parameter omit "Aaron Brady" <castironpi(a)comcast.net> wrote: > > Actually, I wanted a uniform way to call f. f(a,b,c) for both cases if b > can equal <def>. f(a, b=...) #default c f(a, c=...) #default b - Josiah > -----Original Message----- > From: Steven Bethard [mailto:steven.bethard@gmail.com] > Sent: Wednesday, May 09, 2007 3:45 PM > To: Aaron Brady > Cc: python-ideas(a)python.org > Subject: Re: [Python-ideas] parameter omit > > On 5/9/07, Aaron Brady <castironpi(a)comcast.net> wrote: > > Is it possible to signal to a parameter to use its default? > > > > def f( a, b=None, c='' ):... > > > > f( 123, <def>, 'abc' ) > > f( 123, ObjA, <def> ) > > In this case, it's pretty easy:: > > f(123, c='abc') > f(123, ObjA) > > STeVe > -- > I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a > tiny blip on the distant coast of sanity. > --- Bucky Katt, Get Fuzzy > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org http://mail.python.org/mailman/listinfo/python-ideas

1 0

Re: [Python-ideas] the future of the GIL
by tomer filiba 09 May '07

09 May '07

[ moved to python-ideas from python-3000 ] On 5/8/07, Thomas Heller <theller at ctypes.org> wrote: > Wouldn't multiple interpreters (assuming the problems with them would be fixed) > in the same process give the same benefit? A separate GIL for each one? hmm, i find this idea quite interesting really: * builtin immutable objects such as None, small ints, non-heap types, and builtin functions, would become uncollectible by the GC. after all, we can't reclaim their memory anyway, so keeping the accounting info is just a waste of time. the PyObject_HEAD struct would grow an "ob_collectible" field, which would tell the GC to ignore these objects altogether. for efficiency reasons, Py_INCREF/DECREF would still change ob_refcount, only the GC will ignore it for uncollectible objects. * each thread would have a separate interpreter, and all APIs should grow an additional parameter that specifies the interpreter state to use. * for compatibility reasons, we can also have a dict-like object mapping between thread-ids to interpreter states. when you invoke an API, it would get the interpreter state from the currently executing thread id. maybe that could be defined as a macro over the real API function. * the builtin immutable objects would be shared between all instances of the interpreter. other then those, all other objects would be local to the interpreter that created them * extension modules would have to be changed to support per-interpreter initialization. * in order to communicate between interpreters, we would use some kind of IPC mechanism, to serialize access to objects. of course it would be much more efficient, as no context switches are required in the same process. this would make each thread basically as protected as a OS process, so no locks would be required. * in order to support the IPC, a new builtin type, Proxy, would be added to the language. it would be the only object that can hold a cross-reference to objects in different interpreters -- much like today's RPC libs -- only that wouldn't have to work over a socket. * if python would ever have a tracing GC, that would greatly simplify things. also, moving to an atomic incref/decref library could also help. of course i'm not talking about adding that to py3k. it's too immature even for a pre-pep. but continuing to develop that idea more could be the means to removing the GIL, and finally having really parallel python scripts. -tomer

1 0

Re: [Python-ideas] [Python-3000] the future of the GIL
by Leonardo Santagada 09 May '07

09 May '07

Em 09/05/2007, às 06:07, Giovanni Bajo escreveu: > On 07/05/2007 7.36, Josiah Carlson wrote: > >> By going multi-process rather than multi-threaded, one generally >> removes >> shared memory from the equasion. Note that this has the same >> effect as >> using queues with threads, which is generally seen as the only way of >> making threads "easy". If one *needs* shared memory, we can >> certainly >> create an mmap-based shared memory subsystem with fine-grained object >> locking, or emulate it via a server process as the processing package >> has done. >> >> Seriously, give the processing package a try. It's much faster >> than one >> would expect. > > I'm fully +1 with you on everything. > > And part of the fact that we have to advocate this is because > Python has > always had pretty good threading libraries, but not processing > libraries; > actually, Python does have problems at spawning processes: the whole > popen/popen2/subprocess mess isn't even fully solved yet. > > One thing to be said, though, is that using multiple processes > cause some > headaches with frozen distributions (PyInstaller, py2exe, etc.), > like those > usually found on Windows, specifically because Windows does not > have fork(). > > The processing module, for instance, doesn't take this problem into > account at > all, making it worthless for many of my real-world use cases. > -- > Giovanni Bajo Another problem is that althought people like the idea of processing more, the only module on python stdlib is thread and Threading... So why not include processing in stdlib and in both thread and Threading give pointers to its use? This would probably diminish the problem with newcomers on the language that hate the GIL because they don't know better (and also because threading is supported out of the box on python). Yes I know there is fork, but still it is not as useable and pythonic as processing. -- Leonardo Santagada santagada(a)gmail.com

1 0

Re: [Python-ideas] [Python-3000] the future of the GIL
by Leonardo Santagada 09 May '07

09 May '07

Em 09/05/2007, às 04:26, James Y Knight escreveu: > On May 7, 2007, at 1:58 PM, Guido van Rossum wrote: >> As C doesn't have an atomic increment nor an atomic >> decrement-and-test, the INCREF and DECREF macros sprinkled throughout >> the code (many thousands of them) must be protected by some lock. > > I've been intently ignoring the rest of the thread (and will continue > to do so), but, to respond to this one particular point... > > This just isn't true. Python can do an atomic increment in a fast > platform specific way. It need not restrict itself to what's > available in C. (after all, *threads* aren't available in C....) > > Two implementations of note: > > 1) gcc 4.1 has atomic operation builtins: > http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic- > Builtins.html#Atomic-Builtins > > 2) There's a pretty damn portable library which provides these > functions for what looks to me like pretty much all CPUs anyone would > use, under Linux, Windows, HP/UX, Solaris, and OSX, and has a > fallback to using pthreads mutexes: > > http://www.hpl.hp.com/research/linux/atomic_ops/index.php4 > http://packages.debian.org/stable/libdevel/libatomic-ops-dev > > > It's quite possible the overhead of GIL-less INCREF/DECREF is still > too high even with atomic increment/decrement primitives, but AFAICT > nobody has actually tried it. So saying GIL-less operation for sure > has too high of an overhead unless the refcounting GC is replaced > seems a bit premature. > > James This seems like a good idea... how about moving this discussion to python-ideas? -- Leonardo Santagada santagada(a)gmail.com

1 0

subclassing standard module
by Aaron Brady 09 May '07

09 May '07

This on module inspect. I'd like to subclass inspect.BlockFinder, but inspect.getblock uses it directly. Should we encapsulate this?

1 0

PEP for executing a module in a package containing relative imports
by Brett Cannon 07 May '07

07 May '07

Some of you might remember a discussion that took place on this list about not being able to execute a script contained in a package that used relative imports (read the PEP if you don't quite get what I am talking about). The PEP below proposes a solution (along with a counter-solution). Let me know what you think. I especially want to hear which proposal people prefer; the one in the PEP or the one in the Open Issues section. Plus I wouldn't mind suggestions on a title for this PEP. =) ------------------------------------------- PEP: XXX Title: XXX Version: $Revision: 52916 $ Last-Modified: $Date: 2006-12-04 11:59:42 -0800 (Mon, 04 Dec 2006) $ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: XXX-Apr-2007 Abstract ======== Because of how name resolution works for relative imports in a world where PEP 328 is implemented, the ability to execute modules within a package ceases being possible. This failing stems from the fact that the module being executed as the "main" module replaces its ``__name__`` attribute with ``"__main__"`` instead of leaving it as the actual, absolute name of the module. This breaks import's ability to resolve relative imports from the main module into absolute names. In order to resolve this issue, this PEP proposes to change how a module is delineated as the module that is being executed as the main module. By leaving the ``__name__`` attribute in a module alone and setting a module attribute named ``__main__`` to a true value for the main module (and thus false in all others), proper relative name resolution can occur while still having a clear way for a module to know if it is being executed as the main module. The Problem =========== With the introduction of PEP 328, relative imports became dependent on the ``__name__`` attribute of the module performing the import. This is because the use of dots in a relative import are used to strip away parts of the calling module's name to calcuate where in the package hierarchy a relative import should fall (prior to PEP 328 relative imports could fail and would fall back on absolute imports which had a chance of succeeding). For instance, consider the import ``from .. import spam`` made from the ``bacon.ham.beans`` module (``bacon.ham.beans`` is not a package itself, i.e., does not define ``__path__``). Name resolution of the relative import takes the caller's name (``bacon.ham.beans``), splits on dots, and then slices off the last n parts based on the level (which is 2). In this example both ``ham`` and ``beans`` are dropped and ``spam`` is joined with what is left (``bacon``). This leads to the proper import of the module ``bacon.spam``. This reliance on the ``__name__`` attribute of a module when handling realtive imports becomes an issue with executing a script within a package. Because the executing script is set to ``'__main__'``, import cannot resolve any relative imports. This leads to an ``ImportError`` if you try to execute a script in a package that uses any relative import. For example, assume we have a package named ``bacon`` with an ``__init__.py`` file containing:: from . import spam Also create a module named ``spam`` within the ``bacon`` package (it can be an empty file). Now if you try to execute the ``bacon`` package (either through ``python bacon/__init__.py`` or ``python -m bacon``) you will get an ``ImportError`` about trying to do a relative import from within a non-package. Obviously the import is valid, but because of the setting of ``__name__`` to ``'__main__'`` import thinks that ``bacon/__init__.py`` is not in a package since no dots exist in ``__name__``. To see how the algorithm works, see ``importlib.Import._resolve_name()`` in the sandbox [#importlib]_. Currently a work-around is to remove all relative imports in the module being executed and make them absolute. This is unfortunate, though, as one should not be required to use a specific type of resource in order to make a module in a package be able to be executed. The Solution ============ The solution to the problem is to not change the value of ``__name__`` in modules. But there still needs to be a way to let executing code know it is being executed as a script. This is handled with a new module attribute named ``__main__``. When a module is being executed as a script, ``__main__`` will be set to a true value. For all other modules, ``__main__`` will be set to a false value. This changes the current idiom of:: if __name__ == '__main__': ... to:: if __main__: ... The current idiom is not as obvious and could cause confusion for new programmers. The proposed idiom, though, does not require explaining why ``__name__`` is set as it is. With the proposed solution the convenience of finding out what module is being executed by examining ``sys.modules['__main__']`` is lost. To make up for this, the ``sys`` module will gain the ``main`` attribute. It will contain a string of the name of the module that is considered the executing module. A competing solution is discussed in `Open Issues`_. Transition Plan =============== Using this solution will not work directly in Python 2.6. Code is dependent upon the semantics of having ``__name__`` set to ``'__main__'``. There is also the issue of pre-existing global variables in a module named ``__main__``. To deal with these issues, a two-step solution is needed. First, a Py3K deprecation warning will be raised during AST generation when a global variable named ``__main__`` is defined. This will help with the detection of code that would reset the value of ``__main__`` for a module. Without adding a warning when a global variable is injected into a module, though, it is not fool-proof. But this solution should cover the vast majority of variable rebinding problems. Second, 2to3 [#2to3]_ will gain a rule to transform the current ``if __name__ == '__main__': ...`` idiom to the new one. While it will not help with code that checks ``__name__`` outside of the idiom, that specific line of code makes up a large proporation of code that every looks for ``__name__`` set to ``'__main__'``. Open Issues =========== A counter-proposal to introducing the ``__main__`` attribute on modules was to introduce a built-in with the same name. The value of the built-in would be the name of the module being executed (just like the proposed ``sys.main``). This would lead to a new idiom of:: if __name__ == __main__: ... The perk of this idiom over the one proposed earlier is that the general semantics does not differ greatly from the current idiom. The drawback is that the syntactic difference is subtle; the dropping of quotes around "__main__". Some believe that for existing Python programmers bugs will be introduced where the quotation marks will be put on by accident. But one could argue that the bug would be discovered quickly through testing as it is a very shallow bug. The other pro of this proposal over the earlier one is the alleviation of requiring import code to have to set the value of ``__main__``. By making it a built-in variable import does not have to care about ``__main__`` as executing the code itself will pick up the built-in ``__main__`` itself. This simplies the implementation of the proposal as it only requires setting a built-in instead of changing import to set an attribute on every module that has exactly one module have a different value (much like the current implementation has to do to set ``__name__`` in one module to ``'__main__'``). References ========== .. [#2to3] 2to3 tool (http://svn.python.org/view/sandbox/trunk/2to3/) [ViewVC] .. [#importlib] importlib (http://svn.python.org/view/sandbox/trunk/import_in_py/importlib.py?view=mar…) [ViewVC] Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

14 58

Should function attributes be properties?
by Joel Bender 03 May '07

03 May '07

I was expecting this to work: class myprop(property): def __get__(self, obj, cls): print "get", obj, cls def __set__(self, obj, value): print "set", obj, value def f(x): print 'f' f.p f.p = myprop() So it comes out like this: >>> f(2) f get <function f at 0x61270> None Is this a bug? Or was it not in the scope of what properties would be used for? Is this PEP material? Joel

3 2

fixing mutable default argument values
by Chris Rebert 02 May '07

02 May '07

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)

9 25

ordered dict
by Mathias Panzenböck 01 May '07

01 May '07

Some kind of ordered dictionary would be nice to have in the standard library. e.g. a AVL tree or something like that. It would be nice so we can do things like that: for value in tree[:end_key]: do_something_with(value) del tree[:end_key] A alternative would be just to sort the keys of a dict but that's O( n log n ) for each sort. Depending on what's the more often occurring case (lookup, insert, get key-range, etc.) a other kind of dict object would make sense. What do you think? -panzi

5 18