Mailman 3 March 2009 - Python-ideas

A read-only, dict-like optparse.Value
by Thomas Lee 20 Mar '09

20 Mar '09

Hi folks, Would anybody support the idea of read-only dict-like behaviour of "options" for the following code: ==== from optparse import OptionParser parser = OptionParser() parser.add_option("--host", dest="host" default="localhost") parser.add_option("--port", dest="port", default=1234) parser.add_option("--path", dest="path", default="/tmp") options, args = parser.parse_args() ==== As it is, you have to "know" what possible attributes are present on the options (effectively the set of "dest" attributes) -- I often implement something like the following because recently I've had to use command line options in a bunch of format strings: def make_options_dict(options): known_options = ("host", "port", "path") return dict(zip(known_options, [getattr(options, attr) for attr in known_options])) I don't mind having to do this, but having to hard code the options in there feels a bit nasty. Just as useful for my particular use case (i.e passing the options "dict" to a format string) would be something along the lines of options.todict() or dict(options). Even a way to know the set of "dest" attributes that are defined on "options" would be cleaner. e.g. options_dict = dict(zip(options.all(), [getattr(options, attr) for attr in options.all()])) Where options.all() returns all the option "dest" attribute names. Or something to that effect. Any thoughts? Cheers, T

3 3

Re: [Python-ideas] file.read() doesn't read the whole file
by R. David Murray 20 Mar '09

20 Mar '09

Sreejith K <sreejithemk(a)gmail.com> wrote: > I'm using the above codes in a pthon-fuse's file class's read > function. The offset and length are 0 and 4096 respectively for my > test inputs. When I open a file and read the 4096 bytes from offset, > only a few lines are printed, not the whole file. Actually the file is > only a few bytes. But when I tried reading from the Interactive mode > of python it gave the whole file. > > Is there any problem using read() method in fuse-python ? > > Also statements like break and continue behaves weirdly in fuse > functions. Any help is appreciated.... If you think break and continue are behaving differently in a python program that is providing a fuse filesystem implementation, then your understanding of what your code is doing is faulty in some fashion. The fact that python is being called from fuse isn't going to change the semantics of the language. So I think you need to do some debugging to understand what's actually going on when your code gets called. As someone else suggested, if you are perceiving that the data read is short because of what you see at the os level when reading the data from the fuse-plus-your-application filesystem, that is after your python code returns the data to the fuse infrastructure, then that is probably where your problem is and not in the python read itself. (I'm assuming here that the read in question is taking place in a python method called from fuse and is reading real data from a real file...if that assumption is wrong and you are actually reading from a file _provided through_ fuse, then you need to look to your fuse file system implementation for answers.) -- R. David Murray http://www.bitdance.com

1 0

Interactive trace function
by Gerald Britton 20 Mar '09

20 Mar '09

Years ago I worked for a while in REXX. It was a fairly advanced scripting language for its time and we used it on some substantial mainframe projects. One thing that I liked and miss in Python is the "trace" statement. Basically, you could insert a "trace" statement in a REXX program and then, when you ran it, it would step through the program a line at a time. This gave you the chance to follow your code through tricky sections and display variables after each step. I know it saved me hours of troubleshooting of tricky problems. I would like to know if anyone has ever proposed something similar for Python. It would work something like this: 1. In an interactive session, you could issue a trace command. Thereafter, whatever you ran would be done a step at a time, with a terminal prompt after every statement for you to print anything you like that would help you understand the state of the program at that point. 2. From the command line, you could add a --trace option, or something like it, to ask Python to launch the program interactively with trace enabled, which would work as described above. 3. If you have a problematic piece of code, you could insert a trace statement just before the troubled section. Then when you ran the program, when it came to the trace statement, it would begin an interactive trace at that point as described above. (You would have to start your program from the command line for this to make sense.) Has something like this ever come up before? Is there a way to do this today? -- Gerald Britton

3 4

Revision to yield-from implementation
by Greg Ewing 20 Mar '09

20 Mar '09

I have uploaded a small revision to my prototype implementation of the yield-from statement to fix a small refcounting bug. http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/ -- Greg

1 0

Rough draft: Proposed format specifier for a thousands separator (discussion moved from python-dev)
by Raymond Hettinger 19 Mar '09

19 Mar '09

Motivation: Provide a simple, non-locale aware way to format a number with a thousands separator. Adding thousands separators is one of the simplest ways to improve the professional appearance and readability of output exposed to end users. In the finance world, output with commas is the norm. Finance users and non-professional programmers find the locale approach to be frustrating, arcane and non-obvious. It is not the goal to replace locale or to accommodate every possible convention. The goal is to make a common task easier for many users. Research so far: Scanning the web, I've found that thousands separators are usually one of COMMA, PERIOD, SPACE, or UNDERSCORE. The COMMA is used when a PERIOD is the decimal separator. James Knight observed that Indian/Pakistani numbering systems group by hundreds. Ben Finney noted that Chinese group by ten-thousands. Visual Basic and its brethren (like MS Excel) use a completely different style and have ultra-flexible custom format specifiers like: "_($* #,##0_)". Proposal I (from Nick Coghlan]: A comma will be added to the format() specifier mini-language: [[fill]align][sign][#][0][minimumwidth][,][.precision][type] The ',' option indicates that commas should be included in the output as a thousands separator. As with locales which do not use a period as the decimal point, locales which use a different convention for digit separation will need to use the locale module to obtain appropriate formatting. The proposal works well with floats, ints, and decimals. It also allows easy substitution for other separators. For example: format(n, "6,f").replace(",", "_") This technique is completely general but it is awkward in the one case where the commas and periods need to be swapped. format(n, "6,f").replace(",", "X").replace(".", ",").replace("X", ".") Proposal II (to meet Antoine Pitrou's request): Make both the thousands separator and decimal separator user specifiable but not locale aware. For simplicity, limit the choices to a comma, period, space, or underscore.. [[fill]align][sign][#][0][minimumwidth][T[tsep]][dsep precision][type] Examples: format(1234, "8.1f") --> ' 1234.0' format(1234, "8,1f") --> ' 1234,0' format(1234, "8T.,1f") --> ' 1.234,0' format(1234, "8T .f") --> ' 1 234,0' format(1234, "8d") --> ' 1234' format(1234, "8T,d") --> ' 1,234' This proposal meets mosts needs (except for people wanting grouping for hundreds or ten-thousands), but iIt comes at the expense of being a little more complicated to learn and remember. Also, it makes it more challenging to write custom __format__ methods that follow the format specification mini-language. For the locale module, just the "T" is necessary in a formatting string since the tool already has procedures for figuring out the actual separators from the local context. Comments and suggestions are welcome but I draw the line at Mayan numbering conventions ;-) Raymond

19 106

thoughts on generator.throw()
by Gerald Britton 19 Mar '09

19 Mar '09

Today I was reviewing changes in Python 2.5 and I noticed the generator throw() method for the first time. While thinking about what it does and why, a question arose in my mind: Why is it called "throw"? (Yes, I know that Java and possibly other languages use this keyword!) In Python, we have long had a "raise" statement to raise exceptions. I would have thought that the generator method would have been called "raise" as well. But then I saw that it would have been impossible to implement since "raise" is a Python keyword. *Then* I wondered why "raise" is a keyword and not a function. If it were a function you could use it easily in places where today you cannot: if 'foo' == 'bar' or raise(FooBar): # only proceed if 'foo' equals 'bar' otherwise raise FooBar exception is invalid syntax because raise is not a function. Now, I can get around it: def raise_(exception): raise exception ... if 'foo' == 'bar' or raise_(FooBar): ... I have a similar question about the "assert" statement. It could possibly benefit from being a function instead. Of course, changing this would break lots of code, but maybe not any more than making print a function as in 3.0. Thoughts? -- Gerald Britton

11 11

Customizing format()
by Raymond Hettinger 18 Mar '09

18 Mar '09

I've been exploring how to customize our thousands separators and decimal separators and wanted to offer-up an idea. It arose when I was looking at Java's DecimalFormat class and its customization tool DecimalFormatSymbols http://java.sun.com/javase/6/docs/api/java/text/DecimalFormat.html . Also, I looked at how regular expression patterns provide options to change the meaning of its special characters using (?iLmsux). I. Simplest version -- Translation pairs format(1234, "8,.1f") --> ' 1,234.0' format(1234, "(,_)8,.1f") --> ' 1_234.0' format(1234, "(,_)(.,)8,.1f") --> ' 1_234,0' This approach is very easy to implement and it doesn't make life difficult for the parser which can continue to look for just a comma and period with their standardized meaning. It also fits nicely in our current framework and doesn't require any changes to the format() builtin. Of all the options, I find this one to be the easiest to read. Also, this version makes it easy to employ a couple of techniques to factor-out formatting decisions. Here's a gettext() style approach. def _(s): return '(,.)(.,)' + s . . . format(x, _('8.1f')) Here's another approach using implicit string concatenation: DEB = '(,_)' # style for debugging EXT = '(, )' # style for external display . . . format(x, DEB '8.1f') format(y, EXT '8d') There are probably many ways to factor-out the decision. We don't need to decide which is best, we just need to make it possible. One other thought, this approach makes it possible to customize all of the characters that are currently hardwired (including zero and space padding characters and the 'E' or 'e' exponent symbols). II. Javaesque version -- FormatSymbols object This is essentially the same idea as previous one but involves modifying the format() builtin to accept a symbols object and pass it to __format__ methods. This moves the work outside of the format string itself: DEB = FormatSymbols(comma='_') EXT = FormatSymbols(comma=' ') . . . format(x, '8.1f', DEB) format(y, '8d', EXT) The advantage is that this technique is easily extendable beyond simple symbol translations and could possibly allow specification of grouping sizes in hundreds and whatnot. It also looks more like a real program as opposed to a formatting mini-language. The disadvantage is that it is likely slower and it requires mucking with the currently dirt simple format() / __format__() protocol. It may also be harder to integrate with existing __format__ methods which are currently very string oriented. Raymond

7 19

dict '+' operator and slicing support for pop
by Andrii V. Mishkovskyi 18 Mar '09

18 Mar '09

First of all, this is my first attempt at submitting an idea to python-ideas. So, here it goes. :) 1. Add ability to use '+' operator for dicts I often wonder why list and tuple instances have '+' and '+=' operators but dicts don't? It's not that rare in my code (and code written by others, as it seems) that i have to write: a.update(b) return a I do understand that adding additional magic method may be inappropriate for dict, but I think it would be nice addition to a language. So, my proposal is that: x = a + b would become equivalent to x = dict(a, **b) a += b would become equivalent to a.update(b) And the example I gave before would be translated to: return a + b Note, that there is a difference between these two examples in semantics, the latter one creates a new dict. But that's what user doesn't care about in 99% of use-cases. A very basic implementation in Python: >>> class Dict(dict): ... def __add__(self, other): ... return self.__class__(self, **other) ... def __iadd__(self, other): ... self.update(other) ... return self ... >>> a = Dict(foo=12, bar=14, baz=16) >>> b = Dict(spam=13, eggs=17, bacon=19) >>> a + b {'bar': 14, 'spam': 13, 'bacon': 19, 'eggs': 17, 'foo': 12, 'baz': 16} >>> a += b >>> a {'bar': 14, 'spam': 13, 'bacon': 19, 'eggs': 17, 'foo': 12, 'baz': 16} >>> b {'eggs': 17, 'bacon': 19, 'spam': 13} >>> class Dict(dict): ... def __add__(self, other): ... return self.__class__(self, **other) ... def __iadd__(self, other): ... self.update(other) ... return self ... >>> a = Dict(foo=12, bar=14, baz=16) >>> b = Dict(spam=13, eggs=17, bacon=19) >>> a + b {'bar': 14, 'spam': 13, 'bacon': 19, 'eggs': 17, 'foo': 12, 'baz': 16} >>> a += b >>> a {'bar': 14, 'spam': 13, 'bacon': 19, 'eggs': 17, 'foo': 12, 'baz': 16} >>> b {'eggs': 17, 'bacon': 19, 'spam': 13} Note: if this is ever going to be implemented, then Mapping ABCs will have to implement these methods, which doesn't sound like backwards-compatible to me. :) 2. Ability to use slices in pop This was discussed earlier (4.5 years ago, actually) in this thread: http://mail.python.org/pipermail/python-dev/2004-November/049895.html Even though original request became a full-blown proposal (PEP-3132) and was implemented in py3k, the internal discussion about pop allowing slice as arguments has silenced. There was some positive feedback from Python developers and I think I can provide a patch for this functionality in 2 weeks. Is there still some interest in this? There is nothing really hard, this would be something like this: >>> class List(list): ... def pop(self, index_or_slice): ... ret = self[index_or_slice] ... del self[index_or_slice] ... return ret ... >>> x = List(range(10)) >>> x.pop(slice(1, 4)) [1, 2, 3] >>> x [0, 4, 5, 6, 7, 8, 9] >>> x.pop(5) 8 >>> x [0, 4, 5, 6, 7, 9] Note: some people think that pop returning different list or item depending on what is being passed to pop() is bad or something. I don't see a problem here, because simple some_list[index_or_slice] can also return list or just one item depending on what type `index_or_slice` is. -- Wbr, Andrii V. Mishkovskyi. He's got a heart of a little child, and he keeps it in a jar on his desk.

8 9

Keyword same in right hand side of assignments (rev)
by hwpuschm＠yahoo.de 18 Mar '09

18 Mar '09

Thank you very much for correctly remarking that the "definition" I formulated contradicts the examples I gave and is therefore utterly inadecuate: > It is proposed to introduce a Keyword "same", > to be used in the right hand side of assignments, as > follows: > > "xx = same + 5" or "...DELETED..." synonymous with "xx += 5" > "value = 2*same + 5" synonymous with "value =*2; > value +=5" > "switch = 1 - same" synonymous with "switch *-1; > switch +=1" > "lst = same + [5,6]" synonymous with "lst += [5,6]" > > "lst = [5,6] + same" synonymous with "...DELETED..." > "lst[2] = 1/same" synonymous with "lst[2] **=-1" > > and so on. What I would like is to extend the augmented assignment and make it easy to understand for naive readers. I hope the following literary definition is consistent enough to convey the correct meaning: "whenever it is possible, modify the target IN PLACE according to the right hand side expression. If it is not possible to do such a thing, substitute the target object with an object that is build according to the right hand side expression and subsequently deleted" The following examples should be correct: "xx = same + 5" synonymous with "xx += 5" "value = 2*same + 5" synonymous with "value =*2; value +=5" "switch = 1 - same" synonymous with "switch *-1; switch +=1" "lst = same + [5,6]" synonymous with "lst += [5,6]" "lst[2] = 1/same" synonymous with "lst[2] **=-1" The following examples would be extensions: "lst = [5,6] + same" synonymous with "lst.reverse(); lst.extend([6,5]); lst.reverse()" "inmutable = same*(same+1)" synonymous with "unused=inmutable+1; inmutable*=unused; del unused" There seems to be no really simple expression for the above extensions, and I take that as an indication that the proposed feature could be quite useful.

6 8

Keyword same in right hand side of assignments (rev)
by hwpuschm＠yahoo.de 17 Mar '09

17 Mar '09

Thank you very much for correctly remarking that the "definition" I formulated contradicts the examples I gave and is therefore utterly inadecuate: > It is proposed to introduce a Keyword "same", > to be used in the right hand side of assignments, as > follows: > > "xx = same + 5" or "...DELETED..." synonymous with "xx += 5" > "value = 2*same + 5" synonymous with "value =*2; > value +=5" > "switch = 1 - same" synonymous with "switch *-1; > switch +=1" > "lst = same + [5,6]" synonymous with "lst += [5,6]" > > "lst = [5,6] + same" synonymous with "...DELETED..." > "lst[2] = 1/same" synonymous with "lst[2] **=-1" > > and so on. What I would like is to extend the augmented assignment and make it easy to understand for naive readers. I hope the following literary definition is consistent enough to convey the correct meaning: "whenever it is possible, modify the target IN PLACE according to the right hand side expression. If it is not possible to do such a thing, substitute the target object with an object that is build according to the right hand side expression and subsequently deleted" The following examples should be correct: "xx = same + 5" synonymous with "xx += 5" "value = 2*same + 5" synonymous with "value =*2; value +=5" "switch = 1 - same" synonymous with "switch *-1; switch +=1" "lst = same + [5,6]" synonymous with "lst += [5,6]" "lst[2] = 1/same" synonymous with "lst[2] **=-1" The following examples would be extensions: "lst = [5,6] + same" synonymous with "lst.reverse(); lst.extend([6,5]); lst.reverse()" "inmutable = same*(same+1)" synonymous with "unused=inmutable+1; inmutable*=unused; del unused" There seems to be no really simple expression for the above extensions, and I take that as an indication that the proposed feature could be quite useful.

1 0