Mailman 3 - Python-ideas

Re: [Python-ideas] [Python-Dev] PEP Proposal: Revised slice objects & lists use slice objects as indexes
by Neil Toronto 10 Mar '08

10 Mar '08

Alexandre Vassalotti wrote: > On Sun, Mar 9, 2008 at 7:21 PM, Forrest Voight <voights(a)gmail.com> wrote: >> This would simplify the handling of list slices. >> >> Slice objects that are produced in a list index area would be different, >> and optionally the syntax for slices in list indexes would be expanded >> to work everywhere. Instead of being containers for the start, end, >> and step numbers, they would be generators, similar to xranges. > > I am not sure what you are trying to propose here. The slice object > isn't special, it's just a regular built-in type. > > >>> slice(1,4) > slice(1, 4, None) > >>> [1,2,3,4,5,6][slice(1,4)] > [2, 3, 4] > > I don't see how introducing new syntax would simplify indexing. Likewise. It would simplify looping, though: >>> for i in 1:5: ... print i 1 2 3 4 >>> Since this kind of loop happens frequently, it makes sense to shorten it. Slice objects (and syntax) seem ready-made for that - it wouldn't be *new* syntax, just repurposed syntax. Though Forrest didn't bring this up directly, I've often thought that Python's having both xrange and slice (and in 3000, range and slice) is mostly vestigial. Their information content is identical and their purposes are highly analogous. Unifying them would reduce the number of new concepts for a beginner by one, and these are frequently-used concepts at that. Negative indexes could throw the idea for a loop, though. (Pun! Ha ha!) And this makes the colons look like some kind of enclosure: >>> for i in :5: ... Neil

2 1

An official complaint regarding the marshal and pickle documentation
by Aaron Watters 06 Mar '08

06 Mar '08

I just checked the python site documentation on marshal and pickle and I consider them to be irresponsibly and dangerously misleading. For example. Suppose Mercurial is implemented using pickle.load (I sure hope it isn't -- is it?). 1) I send someone a "patch" for their software claiming it makes their package run faster. 2) That person uses mercurial to "unpack" the patch and mercurial uses pickle.load. BAM! That person's filesystem is GONE! AND I'M NOT ASSUMING THAT THERE IS ANY BUG IN MERCURIAL! Now: suppose Mercurial is implemented using marshal: no such scenario is possible unless there is a security bug in mercurial where they explicitly execute something. RESOLVED: pickle should come with a large red label: WARNING: LARK'S VOMIT -- NEVER USE PICKLE TO IMPLEMENT UNTRUSTED ARCHIVING OF ANY KIND. It doesn't have one. Marshal needs no such label: but it has one: *Warning:* The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. This is bullshit. Sorry, for the french and the caps, but this is REALLY IMPORTANT. -- Aaron Watters

12 18

Re: [Python-ideas] adding a trim convenience function
by Erick Tryzelaar 06 Mar '08

06 Mar '08

On Wed, Mar 5, 2008 at 1:30 PM, Matthew Russell <matt.horizon5(a)gmail.com> wrote: > Of couse that should of read: > sites = list(item.strip("http://") for item in sites) {l,r,}strip doesn't actually do what I'm talking about, which confused me for a long time. Consider this simple case: >>> 'abaaabcd'.lstrip('ab') 'cd' ltrim in this case would produce 'aaabcd'. On Wed, Mar 5, 2008 at 1:33 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > >>> [site.replace('http://', '')for site in sites] > ['www.google.com', 'python.org', 'www.yahoo.com'] Unfortunately that would break down in certain cases with a different strings, like 'foo bar foo'.replace('foo', ''), which just results in ' bar '. > Try another use case. I think str pretty well has the basic tools needed > to construct whatever specific tools one needs. Oh sure it can, considering that I can implement ltrim in three lines. This is just to reduce a common pattern in my code, and to remove rewriting it in multiple projects.

3 3

adding a trim convenience function
by Erick Tryzelaar 05 Mar '08

05 Mar '08

I find that when I'm normalizing strings, I end up writing this a lot: sites = ['www.google.com', 'http://python.org', 'www.yahoo.com'] new_sites = [] for site in sites: if site.startswith('http://'): site = site[len('http://'):] new_sites.append(site) But it'd be much nicer if I could use a convenience function trim that would do this for me, so I could just use a comprehension: def ltrim(s, prefix): if s.startswith(prefix): return s[len(prefix):] return s sites = ['www.google.com', 'http://python.org', 'www.yahoo.com'] sites = [ltrim(site, 'http://') for site in sites] Would there be any interest to add this helper function, as well as an "rtrim" and "trim", to the str class? -e

4 4

new super redux (better late than never?)
by Anthony Tolle 05 Mar '08

05 Mar '08

I was looking at the reference implementation in PEP 3135 (New Super), and I was inspired to put together a slightly different implementation that doesn't fiddle with bytecode. I know that the new super() in python 3000 doesn't follow the reference implementation in the PEP, but the code intrigued me enough to offer up this little tidbit, which can be easily be used in python 2.5. What I did was borrow the idea of using a metaclass to do a post-definition fix-up on the methods, but added a new function decorator called autosuper_method. Like staticmethod or classmethod, the decorator wraps the function using the non-data descriptor protocol. The method wrapped by the decorator will receive an extra implicit argument (super) inserted before the instance argument (self). One caveat about the decorator: it must be the first decorator in the list (i.e. the outermost wrapper), or else the metaclass will not recognize the wrapped function as an instance of the decorator class. I think this implementation strikes me as more pythonic than the spooky behavior of the new python 3000 super() built-in, and it is more flexible because of the implicit argument design. This allows things like the ability to use the super argument in inner functions without worrying about the 'first argument' assumption of python 3000's super(). The implementation follows, which is also called autosuper in deference to the original reference implementation. It includes a demonstration of some of its flexibility: ------------------------------------------------------------ #!/usr/bin/env python # # autosuper.py class autosuper_method(object): def __init__(self, func, cls=None): self.func = func self.cls = cls def __get__(self, obj, type=None): # return self if self.cls is not set yet if self.cls is None: return self if obj is None: # class binding - assume first argument is instance, # and insert superclass before it def newfunc(*args, **kwargs): if not len(args): raise TypeError('instance argument missing') return self.func(super(self.cls, args[0]), *args, **kwargs) else: # instance binding - insert superclass as first # argument, and instance as second def newfunc(*args, **kwargs): return self.func(super(self.cls, obj), obj, *args, **kwargs) return newfunc class autosuper_meta(type): def __init__(cls, name, bases, clsdict): # set cls attribute of all instances of autosuper_method for v in clsdict: o = getattr(cls, v) if isinstance(o, autosuper_method): o.cls = cls class autosuper(object): __metaclass__ = autosuper_meta if __name__ == '__main__': class A(autosuper): def f(self): return 'A' # Demo - standard use class B(A): @autosuper_method def f(super, self): return 'B' + super.f() # Demo - reference super in inner function class C(A): @autosuper_method def f(super, self): def inner(): return 'C' + super.f() return inner() # Demo - define function before class definition @autosuper_method def D_f(super, self): return 'D' + super.f() class D(B, C): f = D_f # Demo - define function after class definition class E(B, C): pass # don't use @autosuper_method here! The metaclass has already # processed E, so it won't be able to set the cls attribute def E_f(super, self): return 'E' + super.f() # instead, use the extended version of the decorator E.f = autosuper_method(E_f, E) d = D() assert d.f() == 'DBCA' # Instance binding assert D.f(d) == 'DBCA' # Class binding e = E() assert e.f() == 'EBCA' # Instance binding assert E.f(e) == 'EBCA' # Class binding ------------------------------------------------------------ P.S. I know that using the word 'super' as an argument name might be frowned upon, but I'm just copying what I've seen done in the standard python library (e.g. using 'list' as a local variable name :). Anyway, it doesn't really hurt anything unless you wanted to call the original super() built-in from the decorated method, which would kind of defeat the purpose. P.P.S. Something like this might have been offered up already. I've been searching the mail list archives for a while, and found a few reference to using decorators, but didn't find any full implementations. This implementation also has the advantage of being compatible with existing code.

2 2

List-comprehension-like extensions in normal for loops
by Adam Atlas 03 Mar '08

03 Mar '08

How about letting ordinary for loops contain multiple "for"s and optionally "if"s like in list comprehensions and generator expressions? For example, on a site I was just looking at, it had the line: for record in [ r for r in db if r.name == 'pierre' ]: which could instead be: for record in db if record.name == 'pierre': And of course there could be much more complex ones too. I know you could just do "for x in <some generator expression>" and get the same effect and roughly the same speed, but I think this looks a lot nicer, and it would make sense to have this sort of consistency across the multiple contexts in which "for" can be used.

3 2

Restartable Threads
by Gabriel Grant 03 Mar '08

03 Mar '08

Hi everyone, Why is it that threads can't be restarted? I hope this is the right place for this discussion. If this has been (or should be) discussed somewhere else, I apologize: my searches for "restart thread" and similar only turned up statements that restarting threads is impossible, which haven't satiated my curiosity. Is there any fundamental reason why this can't (or shouldn't) be done? If not, what would you think of making thread restartability an option? For those who are wondering why I might wish to condemn myself by using threads at all (rather than, say, subprocesses), never mind threads that can be restarted while maintaining state, my use case is as follows: I am performing low-level hardware control through a C API that I have wrapped with ctypes. The main "run"-type C function takes a pointer to a struct in which it stores information about its state. This function blocks while the system is running, but also needs access to the shared memory space, and thus has to be executed in its own thread (I believe - other options welcomed). In order to signal events, the function returns with a signal code. Once the signal has been dealt with, the API specifies that the same C function call be made, passing the pointer to the original struct, so that the system can resume operation where it left off. I'm sure this could be done using a standard thread (although I haven't actually done it) with something like: def myloop(): while not self.ret == 0: self.resume_evt.clear() self.ret = sharedLib.blocking_call(self.c_state_struct) self.signal_evt.set() self.resume_evt.wait() t.Thread(target=myloop) ... do some things ... t.signal_evt.wait() ... deal with the signal ... t.signal_evt.clear() t.resume_evt.set() Or some such ugliness, but it seemed to me that the most natural implementation of such a system would be something more like: class myThread(Thread): def __init__(self): self.c_state_struct = structMaker() Thread.__init__(self) def run(self): self.ret = sharedLib.blocking_call(self.c_state_struct) which would then be executed with: >>> t = myThread() >>> t.start() ... do some other stuff ... >>> t.join() >>> signal_handle(t.ret) # deal with the returned value >>> t.start() # resume operation However this is impossible, since a thread's start() method can only be called once (as explained in [1], [2] and [3] python2.5 raises an assertion error, although as of rev 55785 this has been changed to a RuntimeError). What I have been unable to find explained, however, is why this should/needs to be the case. To see if I could get around this limitation, I initially hacked this together: class myThread(Thread): def __init__(self): self.i = 1 Thread.__init__(self) def start(self): Thread.__init__(self) Thread.start(self) def run(self): print self.i self.i += 1 return self.i to be used as: >>> t = myThread() >>> t.start() >>> t.join() 1 >>> t.start() >>> t.join() 2 Obviously it is not usable in the general case, since it completely clobbers the thread's internal state through the repeated __init__()s, but one could certainly imagine a more delicate implementation that saves the relevant bits and pieces, while resetting those that need it. With that in mind, I had a look into threading.py and, not immediately seeing any reason this couldn't be done, implemented essentially that functionality. The attached patch is implemented against threading.py from trunk. I've also uploaded a patched copy of my threading.py that can be used with python2.5 to [4], if anyone needs that. In order to maintain complete backward compatibility, I've left the default behaviour to have threads behave as they do today, but by initializing them with "restartable=True", start() can be called repeatedly. For example: class Counter(Thread): def run(self): if not hasattr(self, "count"): self.count = 0 else: self.count += 1 could be used with: >>> t = Counter(restartable=True) >>> t.start() >>> t.join() >>> print t.count 0 >>> t.start() >>> t.join() >>> print t.count 1 If an attempt is made to restart the thread while it is executing, it still raises a RuntimeError, which I think makes sense: >>> t = LongThread(restartable=True) >>> t.start() >>> t.start() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "threading.py", line 441, in start raise RuntimeError("thread already started") RuntimeError: thread already started So this _seems_ to work, but I have to admit, I'm somewhat afraid to use it. I can't help but wonder: is it safe, or is it tempting the Gods of parallelism to inflict sudden, multi-threaded death? Less superstitious opinions than my own would be greatly appreciated. Thanks, -Gabriel Note: In addition to the patch, I have attached a few usage examples/test cases, that I should really make into actual unit tests. Some of these are expected to fail, so the file can't be executed directly - the examples should be run in an interpreter. [1]: http://docs.python.org/lib/thread-objects.html [2]: http://mail.python.org/pipermail/python-list/2006-November/415503.html [3]: http://lookherefirst.wordpress.com/2007/12/20/can-i-restart-a-thread-in-pyt… [4]: http://ieeesb.mcmaster.ca/~grantgm/reThread/threading.py

4 4

simpler super() syntax
by Arne Babenhauserheide 22 Feb '08

22 Feb '08

Hi, I just spent some time figuring out how and why super needs to be called with *args and **kwds in any class, when I use multiple inheritance (or when some subclass wants to use it), and I got the impression, that simply every class should take *args and **kwds and that super should be called inside the init of every class. Would it make sense to make the init of any class take *args and **kwds implicitely? With that, arguments and keywords would always be passed on (the behaviour we need as soon as we use any multiple inheritance) and the code would look cleaner (I think). At the moment the code for a class with MI looks like this: class Blah(Blubb): def __init__(*args, **kwds) super(Blah, self).__init__(*args, **kwds) with implicit *args and **kwds, it would look like this: class Blah(Blubb): def __init__() super(Blah, self).__init__() And by calling super, I implicitely say, that i want to pass on any leftover args or kwds which (to my knowledge) I must do anyway, since else I am in danger of getting MI bugs. What do you think? Best wishes, Arne -- Unpolitisch sein Heißt politisch sein Ohne es zu merken. - Arne Babenhauserheide ( http://draketo.de ) -- Weblog: http://blog.draketo.de -- Mein öffentlicher Schlüssel (PGP/GnuPG): http://draketo.de/inhalt/ich/pubkey.txt

3 2

exit as a statement
by BJörn Lindqvist 20 Feb '08

20 Feb '08

An idea I have thought about for a while and it makes sense to me... $ python Python 2.4.2 (#1, Oct 13 2006, 17:17:08) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> exit 'Use Ctrl-D (i.e. EOF) to exit.' Argh! Do what I mean, stupid Python! And it is Ctrl+Z on Windows, not Ctrl-D. So exit could be a statement that does the same thing that sys.exit() does currently. Bare "exit" to terminate with return code 0, and "exit X" to terminate with return code X. -- mvh Björn

7 8

Idea for the mapping API
by Raymond Hettinger 10 Feb '08

10 Feb '08

Perhaps, the dict() constructor should accept any object with a __dict__ method. The principle is that almost any object that has key and value pairs (named tuples for example) should be readily convertable to and from a dictionary. The current, non-uniform alternative is for the object to provide an asdict() method and for the user to call it directly. Raymond

5 5