Mailman 3 July 2009 - Python-ideas

Re: [Python-ideas] universal set object for use in set manipulation
by Terry Jones 23 Jul '09

23 Jul '09

See also this thread: http://mail.python.org/pipermail/python-dev/2006-May/064977.html I implemented some of what I was proposing there (in Python though, not C), and am happy to send people the code if there's any interest. These things can't be 100% clean (e.g., what happens when you call len on an infinite set), but you can still get a lot of useful functionality if what's provided matches your use case. Having efficient support for very large sets via complements, having a Universal set, etc., would all be nice. Anyway, see the above thread for more discussion / opinions. Terry

2 1

universal set object for use in set manipulation
by Andy Kish 23 Jul '09

23 Jul '09

Hi all, I think the addition of a universal set object would be a nice touch for python sets. Manipulation of collections of sets becomes much easier with it. Right now, certain operations are easy because empty sets are easy to create while the dual operation requires special cases: set_union = set() for s in sets: set_union |= s # this doesn't work set_intersection = set(???) for s in sets: set_intersection &= s Instead you have to do something contorted like: if len(sets) == 0: set_intersection = set() else: sets_i = iter(sets) set_intersection = sets_i.next() for s in sets: set_intersection &= s Implementation of a universal set would be pretty trivial. Trails of EasyExtend [1] has an implementation (albeit used a bit differently) that's basically the entirety of what's needed. The above intersection case would end up looking something like: set_intersection = set.universal() for s in sets: set_intersection &= s Thoughts? I'm happy to write the patch myself if people like this idea. Thanks, Andy Kish. [1] http://fiber-space.de/wordpress/?p=322

1 0

Re: [Python-ideas] Experiment: Adding "re" to string objects.
by Nick Coghlan 22 Jul '09

22 Jul '09

<switching lists from python-dev to python-ideas> Michael Foord wrote: > MRAB wrote: >> [snip] >> Why not drop the ".re" part? You would, however, then need a new name >> for the re split, eg "re_split". >> >> Or you could make the string the pattern, eg r'whatever(.*)'.match(s). > > +1 for re support built-in to strings. The idea of adding a mutable attribute to strings (even if it plays no part in string equality) sends shivers down my spine. It also seems like it would be difficult to do this in a way that didn't increase the size of all strings by at least one pointer slot. However, the idea of adding more convenience classes to the re module may still have some merit. In particular, when checking against multiple regexes, being able to do something like the following might be helpful: m = re.Matcher(s) if m.match(r'whatever(.*)'): print m.group(1) elif m.match(r'something (.*) else(.*)'): print m.group(2) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

10 18

2c about package directories
by Tennessee Leeuwenburg 22 Jul '09

22 Jul '09

Hi all, Apologies if any of this has been said before. I've been loosely following the ongoing discussions about packaging utilities, plus a few people's experiences with attempting to manage their own Python environments. I was thinking about a potential directory layout approach which might be attractive to some people and which I think could be useful for automatic packaging. /usr/local/bin (or whatever the system prefix is) python --> 'standard python version' python2.6 --> 'standard python 2.6' python2.7 python3.1 etc /usr/local/lib/python (or whatever the system prefix is) /python2.6 /core /site-packages /applications /application-name-packages /python2.7... etc /home/username/.python/ /python2.6 /user-packages /applications /application-name-packages With that layout, it should be possible for: -- Anyone to always be able to run a system python installation with 'core' libraries only -- Run a system python installation with 'sitewide' libraries -- Deploy application-specific libraries to a system python installation

1 0

Proposal for extending the collections module - bags / multisets, ordered sets / unique lists
by Michael Lenzen 21 Jul '09

21 Jul '09

In a nutshell, I want to add 2 new classes (and respective frozen varients) to the collections module, namely a bag (multiset) class and a setlist (ordered set/unique list) class. I know this has been floated before, but I think it merits more consideration. Everything I'm writing here is available with better formatting here: http://code.google.com/p/python-data-structures/wiki/collections The latest version of my implementation is here: http://python-data-structures.googlecode.com/svn/trunk/collections_.py I look at the basic data structures of Python like this (ignoring mutability which I will mention later) : | Unordered | Ordered | ----------------------------------- Non-Unique | | list | ----------------------------------- Unique | set | | ----------------------------------- Python is missing 2 basic collections. First, an unordered non-unique collection, a bag aka multiset, the simplest collection there is. You can't even really call them a data structure because there's no structure to them (which is their beauty). Secondly it's missing an ordered unique collection, an ordered set or unique list, what I'm calling a setlist. That is the scope of when I started out, of course it grew and grew. The last addition but the first I will list, because I think it is the coolest and provides the most functionality, is the new function I propose adding to collections. The function is simply called 'collection' and has the following signature: collection(iterable=None, mutable=False, ordered=False, unique=False) What it does is return a collection (instantiated from iterable if passed) with the passed properties. This will only be possible with then new classes I propose, because they don't exist yet. New ABCs (abstract base classes) -------------------------------- After starting to study the ABCs built in to Python I came up with 2 more ABCs that I think should be included: Collection - implements Sized, Iterable and Container. Sequence, Set and Mapping would then all inherit Collection instead of the three aforementioned classes. Built-in classes that would be subclasses of Collection are set, frozenset, list and tuple. Collection would provide 2 methods: * _from_iterable(cls, it) - This is a classmethod that I took from Set, for convenience of returning new instantiations * __getitem__(key, value) - This is an abstract method. Every Collection would have to make its elements accessible. Of course set and frozenset do not currently implement this method so they will have to be changed, more later. Mutable - I think there should be one metaclass for all mutable types just like there is currently Iterable, Sized and Container. We could then check instanceof(obj, Mutable). MutableSequence, MutableSet and MutableMapping would inherit Mutable, or we might be able to just do away with them altogether. There are 3 abstractmethods defined: * pop(self) - This is the simplest to implement, all mutable classes already do. * __setitem__(self, key, value) - This is already implemented for list and dict but would have to be defined for set, more later * __delitem__(self, key) Same as above. Extending set to fit the ABCs ----------------------------- I mentioned before that sets don't currently fit into my model. Here are my propositions for the three methods that would need to be implemented. * set.__setitem__(self, elem, value) - Set whether or not elem is in the set based on the boolean evaluation of value. s[elem] = 1 would call s.add(elem) and s[elem] = False would call s.remove(elem), appropriately raising a KeyError if elem not in s. The reason I chose this way is that the only thing you can set about an element of a set is whether or not it is present. * __getitem__(self, item) - Simply returns item in self. Just like a bag returns the multiplicity of an item when you get it, this returns the number of times item appears in self. This also corresponds to the definition of setitem. * __delitem__(self, item) - This is the simplest case, just make it an alias for set.remove(item) I think this is the most controversial part of my proposition, but the more I think about it, the more I like it. []s aren't reserved for one single purpose in Python, lists restrict the index to an integer while dicts allow any Hashable object. list allows slicing inside the []s while it is meaningless for a dict. Finally, the New Classes ------------------------ These are the 2 new Collections (4 if you count the frozen variants) to fill in the holes I mentioned in the beginning. bags - These would be true multisets, AKA unordered tuples. See the wikipedia page on Multisets for more info. The benefits of a bag vs a list are that order doesn't matter for equality testing and that searching for an item is constant time. The latter is the real convincer for me. I think this should replace the Counter class in the collections module which doesn't provide anything more than defaultdict(int) does. setlist - AKA unique list, ordered set, bijection int->hashable, and ordered collection of unique elements. I find myself often creating a list of unique elements, then an inverse dict to look up the location quickly or test inclusion. I chose the name setlist because it's not a set xor a list, it's both. Then it occurred to me that bands' setlists are a perfect example of this data structure, an ordered collection of unique elements, real bands that is, http://en.wikipedia.org/wiki/Vertigo_Tour#The_encores -Michael Lenzen

14 42

Proposal for function expressions
by Chris Perkins 20 Jul '09

20 Jul '09

I have a proposal for a language feature - a limited form of function- definition expression, similar (superficially) to Ruby's blocks. The big problem with making "def" an expression is the indentation of the function body. You can't just embed an indented block of code into the middle of an expression. The solution I propose (inspired by Ruby) is to allow a block of code to be appended to the end of an expression in certain, limited circumstances. The block is simply syntactic sugar for a local anonymous function definition which is passed as an argument to the function call that it follows. First, a simple example to give the broad strokes: foo() do: BODY is equivalent to: def ANON(): BODY foo(ANON) where the name ANON is a placeholder for illustration - it is never actually bound in the local namespace. Specifically, here is what I propose: * A new syntactic construct, the "block", which consists of: * the keyword "do" (alternative: "def", repurposed) * an optional argument list * a colon * an indented block * A block is only allowed immediately following a call (or indexing expression? see below), on the same line as the closing right brace * The block defines an anonymous local function which is passed as an argument to the call that it follows. * The block is passed as the last argument to the function by default, but this can be changed by putting a placeholder, the character "&", into the parameter list of function. This is best illustrated by example: # 1) In the following, foo recieves TWO arguments: 23 and a callable, in that order. foo(23) do: pass # 2) Here, foo also recieves TWO arguments: a callable and 23, in that order. foo(&, 23) do: pass Why "&"? No particular reason - just because Ruby uses "&" for something similar (but not the same). Why do we need the "&" feature? Consider this example: map(&, my_list) do(item): return do_something_to(item) This also works for keyword arguments: foo(a=3, b=&, c=7) do(): whatever() To make this syntax work, we need several restrictions on where a block is allowed. Intuitively, the rules are: * If the line following "do" would normally be indented, then a block is not allowed. * If the line following the "do" would be one on which leading indentation is insignificant, a block is not allowed. To clarify, the first rule means that this is not allowed: if foo() do: # are we in the body of the block, or of the if? The second rule means that this is not allowed: bar(23, foo() do: body_of_block() ) # closing brace of call to bar Here are some properties of blocks that may not be immediately obvious: * Blocks are a feature of the call site, and do not affect function definitions. In other words, there is no such thing as a "function expecting a block", as there is in Ruby. From the callee's point of view, he has simply been passed a callable as an argument. * Blocks can be used with any existing callable that expects to be passed a function as one of its arguments. OK, let's move on to the fun part: Motivating Examples. ################### # Networking dfr = twisted.whatever(...) dfr.addCallback() do(result): handle(result) dfr.addErrback() do(err): handle_err(err) ################### # GUI (for some hypothetical library) b = my_widget.add_button('Go') b.on('click') do(evt): if evt.ctrl_key: do_something() else: do_other_stuff() ################### # Routing HTTP requests map.match('/') do(req): return render_home(req) map.match('/about') do(req): return render_about(req) map.match(/(?P<controller>\w+)\/$/) do(req, **kw): return handlers[kw['controller']](req) ################### # Threads thread.start_new() do: do_some_work() ################### # Reduce the temptation to cram too much into a lambda. # Sort a list of filenames that look like this: # Xyz-1.1.3.tgz, acbd-4.7.tgz, Xyz-1.2.5.tgz, ... my_list.sort(key=&, reverse=True) do(item): name, _ = os.path.splitext(item) root, _, version = name.partition('-') parts = version.split('.') return (root.lower(),) + tuple(map(int, parts)) stuff = filter(&, my_list) do(item): # some code here... return pred(item) stuff = re.sub(rx, &, s) do(m): t = m.group(0) if cond(t): return t.upper() else if othercond(t): return t.lower() else: return '' ################### # Faking switch switch(expr) do(case): case(int) do: stuff() case(str, bytes) do: other_stuff() switch(value) do(case): case[0:10] do(val): handle_small(val) case[10:100] do(val): handle_medium(val) case.default() do(val): handle_big(val) ################### # Overloaded functions (adapted from PEP 3124) @overload def flatten(when): when(basestring) do(ob): yield ob when(Iterable) do(ob): for o in ob: for ob in flatten(o): yield ob when.otherwise() do(ob): yield ob # later: flatten.when(file) do(f): for line in f: yield line ################### # Sort-of a "with(lock)" block, but running the body in a thread. pool.run_with(my_lock) do: # lock is held here work() other_work() # in parallel ################### Well, that should give you the general idea. Some final random points: * As proposed, blocks are a new kind of beast in Python - an indented block that is part of an expression, rather than being a statement. I think that by constraining a block to be the last thing in an expression, this works out OK, but I may be missing something * Since writing this, I have started to lean towards "def" as the keyword, rather than "do". I used "do" in the examples mostly to make them look like Ruby, but I suspect that making Python "look like Ruby" is not a design goal for most people :) * Terminology: "block"? "def expression"? "anonymous def"? "lambda++"? * Idea: Could we relax the rules for where a block is allowed, removing the constraint that it follows a call or indexing expression? Then we could do, for example, this: def f(): return def(args): pass * Would a proliferation of nameless functions be a nightmare for debugging? Implementation is here: http://bitbucket.org/grammati/python-trunk/ Patch is here: http://bugs.python.org/issue6469 To be honest, I'm not entirely sure if the patch is in the right format - it comes from hg, not svn. Chris Perkins

24 80

Re: [Python-ideas] Make return inside a finally a SyntaxError
by Terry Jones 19 Jul '09

19 Jul '09

I agree that having finally subsume returns and exceptions in the try block will at first be surprising to people, and that it can lead to easy-to-create but difficult-to-find bugs. But I think the current behavior is definitely for the best: - It's dead simple. The finally clause is always done last, and is always responsible for the return of the function *no matter what* may have happened in the try clause. - If finally were to automatically propagate any exception from the try clause, how would you turn that off when you needed/wanted to? You'd soon wind up wanting another keyword, perhaps seriouslyfinally, that was guaranteed to be run at the very end. If finally propagated exceptions, what should it do if there's an exception raised in the finally block? It begins to get complicated. The primitive blanket solution is better, IMO. - Besides, if you do want a finally clause to propagate an exception from its try clause, that's simple as you can just store the exception somewhere in an except clause and then let the finally detect it and re-raise it, if appropriate. By keeping finally's semantics simple and resisting the temptation to make it more sophisticated (e.g., propagating exceptions or return values) you wind up with a better tool for the programmer who does know what they're doing. The more sophisticated behavior can easily be added as you need it on a case-by-case basis. The added functionality shouldn't be provided unconditionally by the language because you then can't disable it. I.e., I prefer the control I get from the current semantics of finally. Terry

6 5

Make return inside a finally a SyntaxError
by Michael 18 Jul '09

18 Jul '09

Here are two examples of why allowing return inside a finally block is a bad idea: def f(): try: return 3 finally: return 4 def f(): try: raise Exception() finally: return 4 Michael Foord -- http://www.ironpythoninaction.com

17 32

Augment dis.dis to autocompile strings
by Terry Reedy 18 Jul '09

18 Jul '09

Currently "dis.dis(x=None) Disassemble the x object. x can denote either a module, a class, a method, a function, or a code object. " The behavior depends on the class of x, so dis already has an internal switch. Currently, to see the assembly equivalent of CPython bytecode for a small snippet, one must either wrap it in a function or *remember* the exact idiom: compile("while True: break", "", "exec") Proposal: if the input is a string, do the above (do what I mean) instead of raising TypeError: don't know how to disassemble str objects Any negatives before I submit a feature request? Terry Jan Reedy

4 6

Re: [Python-ideas] Make return inside a finally a SyntaxError
by Terry Jones 18 Jul '09

18 Jul '09

>>>>> "Michael" == Michael Foord <fuzzyman(a)gmail.com> writes: Michael> The bit that has to be *added* in Python (and usually isn't added Michael> because it isn't at all obvious) is: Michael> Exceptions are not reraised if the finally returns or breaks. OK. >> If finally propagated exceptions, Michael> It does. Sorry - I meant propagates the try exception even in the case that the finally does a return, break, or raises itself. >> It begins to get complicated. Michael> No, it gets simpler - no special cases for the exception handling Michael> of finally in the presence of a return. I think it's more complex. Either, or both, of the try block and the finally block may return/raise etc. What to do when those things conflict? Which block's opinion has priority? It seems clear (to me) that the finally block should have the final word, no matter what - including (perhaps even especially) if its ultimate conclusion conflicts with what the try block did. Terry

1 0