Python-ideas May 2008

python-ideas@python.org

22 participants
9 discussions

Ideas towards GIL removal
by Greg Ewing 18 Jul '08

18 Jul '08

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". Any refcount operation on a permanent object would be a no-op, so no locking would be needed. This would also have the benefit of eliminating any need to write to the object's memory at all when it's only being read. 2) Objects owned by a thread Python code creates and destroys temporary objects at a high rate -- stack frames, argument tuples, intermediate results, etc. If the code is executed by a thread, those objects are rarely if ever seen outside of that thread. It would be beneficial if refcount operations on such objects could be carried out by the thread that created them without locking. To achieve this, two extra fields could be added to the object header: an "owning thread id" and a "local reference count". (The existing refcount field will be called the "global reference count" in what follows.) An object created by a thread has its owning thread id set to that thread. When adjusting an object's refcount, if the current thread is the object's owning thread, the local refcount is updated without locking. If the object has no owning thread, or belongs to a different thread, the object is locked and the global refcount is updated. The object is considered garbage only when both refcounts drop to zero. Thus, after a decref, both refcounts would need to be checked to see if they are zero. When decrementing the local refcount and it reaches zero, the global refcount can be checked without locking, since a zero will never be written to it until it truly has zero non-local references remaining. I suspect that these two strategies together would eliminate a very large proportion of refcount-related activities requiring locking, perhaps to the point where those remaining are infrequent enough to make GIL removal practical. -- Greg

8 19

Proposal to add new built-in struct (was: Add kwargs to built-in function object)
by Brandon Mintern 02 Jun '08

02 Jun '08

This is a proposal to add a new built-in named struct: struct(**kwargs) Return a struct object which has the attributes given in kwargs. The name is really unimportant, and I'm open to other ideas, but I feel that many people have a good idea of what a struct is, and accessing an object returned from struct would look identical to the access of a struct in C, so it seems appropriate to me. The rationale: It is often helpful to package related information together in order to make passing the information to various functions more convenient. The easiest ways (currently) to package such information is to put it into a tuple (the easiest) or a dict. Putting the information into a tuple may be easy initially, but it has high costs later on as it becomes hard to remember what order the information is in, and to someone reading the code, the intent of the code is far from clear. Putting the information in a dict is a bit harder than a tuple, but since it is more readable later on than a tuple, this method is often used. Still, the access pattern is more cumbersome than it could be; foo["bar"] is more cumbersome than, say, foo.bar. This is especially the case if you have a dict or list of foos, where you then have to use foos[i]["bar"]. Both tuple and dict solutions suffer down the line when the information gets to be complicated enough to warrant a class of its own. It involves changing both the spot in the code where the information is created (to use the new class constructor), as well as changing every single field access in the code (changing every foo[0] or foo["bar"] to foo.bar). An alternative is to use NamedTuple, NamedDict, NamedList, or to create your own class. As long as these are more complicated to use than a tuple or a dict, however, they are not likely to be used for this purpose. Another problem is that all of these methods require you to go to the trouble of thinking of a name for your class, and if you later decide to add more information to your packaged object, you have to make two changes (in the list of attributes / constructor and in the place where you instantiate your object). Enter struct. Using struct is intended to be just as easy as using a dict (actually, easier when the number of fields is more than two or three), and not much harder than using a tuple. To declare a struct foo with attribute bar, you simply use: foo = struct(bar="barvalue") Access becomes very easy and readable: foo.bar Adding new fields is as easy as changing the initial instantiation, in one place: foo = struct(bar="barvalue", baz="bazvalue") Later on down the line, when you decide that you are doing more with foo than a struct should be doing, you can easily define a class Foo which inherits from struct, and since accesses to foo already look like foo.bar, you only have one spot in the code to change: foo = Foo(bar="barvalue", baz="bazvalue") and the rest "just works" with no changes. The implementation: class struct (object): def __init__ (self, **kwargs): self.__dict__.update(kwargs) def __repr__ (self): """ Using self.__class__.__name__ allows classes to inherit from struct and automatically have a nice __repr__ method. """ return "%s(%s)" % (self.__class__.__name__, ", ".join("%s=%s" % (attr, repr(val)) for attr, val in self.__dict__.iteritems()),) # or .items() # in Python 3K def __str__ (self): return self.__repr__() def __eq__ (self, other): """ Implements comparison operation mirroring that of a C struct. """ return self.__dict__ == other.__dict__ def __ne__ (self, other): """ See note for __eq__. """ return not self.__eq__(other) def __setattr__ (self, name, value): """ I think it makes the most sense for a struct to have immutable fields. As soon as you start to add more fields, you should be using something other than a struct. """ if name in self.__dict__: self.__dict__[name] = value else: raise(AttributeError("'struct' object has no attribute '%s'" \ % (name,))) def __len__ (self): """ I'm not sure that it's really necessary to include this, but I could see where it might be helpful in some instances. """ return len(self.__dict__) def __iter__ (self): """ See note for __len__ """ return self.__dict__.itervalues() # or .values() in Python 3K Sample usage: >>> a = struct(one=1, two=2) >>> a struct(two=2, one=1) >>> a.one 1 >>> a.two 2 >>> a.three Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'struct' object has no attribute 'three' >>> a.one = "one" >>> a.one 'one' >>> a.three = 3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "struct.py", line 39, in __setattr__ % (name,))) AttributeError: 'struct' object has no attribute 'three' >>> b = struct(one=1, two=2) >>> b struct(two=2, one=1) >>> a == b False >>> a.one = 1 >>> a == b True >>> len(a) 2 >>> print ", ".join(str(v) for v in a) 2, 1 >>> 1 in a True >>> "one" in a False Ideas or feedback, anyone?

8 17

Function to unnest for-statements
by Carl Johnson 25 May '08

25 May '08

This is based off of http://boredzo.org/blog/archives/2007-10-22/generating-all-combinations . It's also discussed at http://reddit.com/info/5ywrs/comments/ and someone with a lot of spare time can read Knuth's fascile on the subject at http://www-cs-faculty.stanford.edu/~knuth/fasc2a.ps.gz . OK, suppose you have a situation where you need to loop through all combinations of 3 lists. The simple solution is to write three nested for-loops: for x in xs: for y in ys: for z in zs: #Do something Of course, this is obvious O(n^3), which is bad, but sometimes you don't have a choice. However, what if (to use an example I ran into), you're making up a truth table. If you know you have three variables, you might write: for a in [True, False]: for b in [True, False]: for c in [True, False]: print a, b, c Which yields True True True True True False True False True True False False False True True False True False False False True False False False But if you don't know how many variables you'll have, you're stuck writing a complicated function instead of using a nice, simple for-loop. So, going back to the first example, wouldn't it be nicer to write: for x, y, z in combine(xs, ys, zs): #Do something If nothing else, this has the advantage that if you don't have to nest the for-loops, things don't end up being so deeply indented on the screen. Similarly, a general truth table maker can be written: ts_and_fs = [[True, False]] * num_vars for variables in combine(*ts_and_fs): print variables So, I think a "combine" function (or some other, better name) would be a good thing to have, at least in the itertools, if not as a built-in function. How to implement it? Well, probably it should be done in C for efficiency, but of the versions done http://boredzo.org/blog/archives/2007-10-22/generating-all-combinations the one that I like best is def combine(*sequences): #Test to guard against infinite recursion if sequences: def _inner(*seqs): if len(seqs) == 1: for i in seqs[0]: yield (i, ) else: for rest in combine(*seqs[:-1]): for i in seqs[-1]: yield rest + (i, ) return _inner(*sequences) else: #Empty generator that yields StopIteration def _inner(): return yield return _inner() However, I'm also convinced that there is a more efficient way to do this. So, what do you think? Is this a common enough need that it should be built into itertools? The main namespace? Or should we leave it out, since adding it would encourage people writing O(n^x) algorithms? If nothing else, list members can enjoy rewriting this function for fun. -- Carl Johnson

7 8

Queue suggestions
by George Sakkis 23 May '08

23 May '08

I'd like to propose a signature simplification and a new method for the Queue.Queue class: 1) Drop the optional `block` argument from put() and get(), since all the meaningful combinations of (block, timeout) are equivalent to passing block = (timeout is None or timeout>0). IOW, instead of passing block=False pass timeout=0 (or negative). Obviously this is to be considered for 3.x only. 2) Add a new `rotate` method as the atomic equivalent of put_nowait(get_nowait()). Currently I use the following subclass but it would be nice to have it in the base Queue: class RQueue(Queue): def rotate(self, n=1): '''Rotate this queue n steps to the left (if it is not empty). Rotating one step is equivalent to an atomic q.put_nowait(q.get_nowait()). ''' if n < 0: raise ValueError('n must be non-negative') self.mutex.acquire() try: if not self._empty(): self._rotate(n) finally: self.mutex.release() def __init__(self, maxsize=0): Queue.__init__(self, maxsize) if hasattr(self.queue, 'rotate'): # deque has rotate() since v2.5 def _rotate(n): # negative n for left rotate self.queue.rotate(-n) else: def _rotate(n): put,get = self._put, self._get for i in xrange(n): put(get()) self._rotate = _rotate What do you think ? George

2 1

Re: [Python-ideas] Proposal to add new built-in struct (was: Add kwargs to built-in function object)
by Brandon Mintern 23 May '08

23 May '08

On Thu, May 22, 2008 at 6:25 PM, Matthew Russell <matt.horizon5(a)gmail.com> wrote: > How about: > collections.attrs(**kwargs) > or > collections.record(**kwargs) I've used the name "Record" in the past, but when I was trying to figure out if Python had anything like a C struct, I searched specifically for "python struct". I'm not sure how many others do the same, but perhaps using a name like "structure" would be better than "record"? (On a side note, see what I mean about the work involved in naming a class? :-) > I think I prefer the idea of this being a factory that returns an anonymous > object as opposed to a class: > > def record(**kwargs): > class Record(object): > def __init__(self, **kw): > for (k, v) in kwargs.items(): > setattr(self, k, v) > return Record() I'm guessing that the last line was meant to be "return Record(**kwargs)" > This is mainly because a class you later substitude for the > result of record(**kwargs) may later end up wanting to take kwargs as > optional values which you might not want as instance variables, or even > posistional arguements. > > x = MyThing(x=1,y=2, default_spam=False) > > > So you could easilty do: > > x = MyThing(default_spam=False, **record_obj.__dict__) > > Matt I can see where you're coming from, but this change would mean that you would no longer be able to inherit from record (or structure, or whatever). Also, if you're making that many changes to the class, it wouldn't be too hard to rewrite the initial instantiation to not use positional parameters with or without kwargs (after rewriting the constructor, which you would clearly be doing, anyways). Another caveat is that each call to record() will create an object of a different class. Thus, if you have the following: employees = {} for ssn, fn, ln in employee_data_list: employees[ssn] = record(ssn=ssn, first=fn, last=ln) Then employees[ssn1].__class__ != employees[ssn2].__class__. This seems like it could be an issue at some point. Brandon

1 0

Add kwargs to built-in function object
by Brandon Mintern 23 May '08

23 May '08

I would like to propose to change the built-in function "object" to have the following syntax: object(**kwargs) Return a new featureless object. object is a base for all new style classes. It has the methods that are common to all instances of new style classes. If kwargs is given, the returned object's __dict__ will be kwargs (or something to that effect). An example: a = object(one=1, two=2) a.one # returns 1 a.two # returns 2 The justification: On several occasions I have needed a collection of attributes to represent a single item. When this happens, there are really three options: 1. Use a tuple. This works well for temporarily re-packaging items in a loop or for quick-and-dirty code, but it rapidly becomes unreadable and hard to maintain. It is not long before you forget what order the attributes are in, and at first glance, it is not clear what kind of object is being indexed. 2. Use a dict. This is an improvement over tuples on readability, but they can be a pain to build and overly-cumbersome to access later. I understand that dicts are used all over the place in Python, but I still think of them (in the general case) as a map of keys to values where the dict represents a collection, not an object. 3. Use a class. This requires coming up with a name for the class and then writing the class (admittedly, this should be easy). Afterwards, this is the most convenient, readable method for representing the data, but since it requires non-trivial effort up front, this method may be avoided until it's truly apparent that it is necessary. A real-world example: Let's say that I want to have a map of employee SSNs to employee data. I am going to be manipulating this information in various ways, but not really in any ways that justify the use of class methods. At any rate, let's build this data from a file where each line is SSN First Last Salary with the items being whitespace-delimited. The code, then, will be: employees = {} for ssn, first, last, salary in (line.split() for line in open(employee_file)): employees[ssn] = (ssn, first, last, salary) # tuple method employees[ssn] = {"ssn": ssn, "first": first, "last": last, "salary": salary} # dict method employees[ssn] = Employee(ssn, first, last, salary) # assumes class with proper constructor # first name of employee with SSN employees[SSN][1] # tuple method -- quite unreadable employees[SSN]["first"] # dict method -- more readable but sub-optimal employees[SSN].first # class method -- readable and simple Now, because of the advantages the class method offers in terms of readability, I have written a convenience class that makes using it easier: class Record: """ A class intended to provide a simple interface for creating objects with named fields. That is, instead of returning a tuple and indexing it or writing a unique class, you can simply do something like: a = Record(x=1, y=2) a.x # 1 a.y # 2 a # Record(x=1, y=2) """ def __init__ (self, **kwargs): self.__dict__.update(kwargs) def __repr__ (self): return "Record(%s)" \ % ", ".join("%s=%s" \ % field for field in self.__dict__.iteritems()) Now, the line in the for loop above becomes employees[ssn] = Record(ssn=ssn, first=first, last=last, salary=salary) and I have completely avoided the need to define a unique class. Note that as the number of fields increases, this becomes easier to use inline than the dict method, at the same time that it avoids the upfront costs of having to build a new class for every distinct type of object in the program. It turns out that other people find such a method useful as well. According to Catherine Devlin on the centralOH Python list, it is recipe 4.18 from the Python Cookbook (2nd ed.). Several others from the mailing list stated that they had created similar solutions themselves. Thus, my suggestion is to simply build such functionality directly into the language. While scanning the built-in functions page (I don't use any, all, or enumerate nearly enough), I noticed the object() function, and since its purpose is to return a featureless object, it seems to fit the bill quite well. Adding my suggested functionality should break nothing (since the kwargs would be optional) and would allow people to stop baking their own solutions to this common problem, while getting the speed bonus of a C implementation (if it helps). Thus, the code above becomes for... employees[ssn] = object(ssn=ssn, first=first, last=last, salary=salary) employees[SSN].first Does anyone else think this might be a good idea for Python 2.6/3K? Thanks, Brandon

10 16

Removing some string methods?
by A.M. Kuchling 19 May '08

19 May '08

Has any consideration been given to removing some of the lesser-used string methods in 3.0? e.g. .center, .ljust/rjust, .zfill, .swapcase? --amk

4 6

Give generators a name?
by Terry Reedy 15 May '08

15 May '08

I have 2 related proposals: 1. Give generators a .__name__ attribute that is the same as their curent (3.0a5) .gi_code.co_name subattribute. just as funct.__name__ is func.__code__.co_name. My reason is, I expect, much the same as that for func.__name__. I am using the generator name (for bad-iterator-output messages in a test function) and would prefer to get it through a cross-implementation 'public' interface' rather than a cPython internal implementation detail (which I understand code object to be). I am otherwise trying to avoid using cPython internals. (Is there any plan to change the gi_* attributes the way the func_* attributes were?) 2. Whether or not 1 is adopted, add the name to the representation: <gfuncname generator object as..> or <generator object gfuncname at ..> Conceptually, I see a generator function as an abbreviated version of a iterator class, with most of the boilerplate removed, that defines a subclass of the generator class. So I think the subclass name should be part of its representation. Terry Jan Reedy

3 2

decimal by default
by Joshua Spoerri 13 May '08

13 May '08

Should decimal be the default for floating period literals? E.g. 1.2 would actually be decimal.Decimal("1.2") and float(1.2) would be used to get traditional binary float point.

4 4

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas May 2008