Mailman 3 May 2009 - Python-ideas

Heap data type
by Facundo Batista Dec. 7, 2009

Dec. 7, 2009

Using the module heapq, I found that it's not easy to use. Or, at least, that it could be straightforward to use. My main issue is that for correct usage: - user should provide an external list, that shouldn't use for anything else to don't break the invariant - to alter the heap queue, a bunch of functions must be used, passing always the external list I think that changing "external list" for "internal attribute", and "bunch of functions " for "methods", it will leave the module easier to use, safer, and more object oriented. So, I basically coded it. What do you think about including this class in the heap module? """ class Heap(object): '''Maintains a heapified list, always conserving the invariant. Heaps are arrays for which heap[k] <= heap[2*k+1] and heap[k] <= heap[2*k+2] for all k, counting elements from zero. ''' def __init__(self, iterable=None): '''Initializes the heap from any iterable. >>> Heap([1, 2]) Heap([1, 2]) >>> Heap([]) Heap([]) >>> Heap() Heap([]) >>> Heap((1, 2, 3)) Heap([1, 2, 3]) >>> Heap(x**2 for x in range(3)) Heap([0, 1, 4]) ''' if iterable is None: self._queue = [] else: self._queue = list(iterable) heapq.heapify(self._queue) def __repr__(self): return "Heap(%s)" % self._queue def push(self, item): '''Push the item to the heap queue. >>> h = Heap() >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.push(2) >>> h.push(1) >>> h.push(0) >>> h Heap([0, 2, 1, 5, 3, 4]) ''' heapq.heappush(self._queue, item) def pop(self): '''Pop one item from the heap queue. >>> h = Heap([0, 2, 1, 5, 3, 4]) >>> h.pop() 0 >>> h.pop() 1 >>> h.pop() 2 >>> h.pop() 3 >>> h.pop() 4 >>> h.pop() 5 >>> h Heap([]) >>> h.pop() Traceback (most recent call last): ... IndexError: index out of range ''' return heapq.heappop(self._queue) def pushpop(self, item): '''Push the item and pop one from the heap queue. Note that this method is more efficient than calling both push() and pop() separately. >>> h = Heap() >>> h.pushpop(7) 7 >>> h.push(5) >>> h.push(4) >>> h.push(3) >>> h.pushpop(7) 3 >>> h.pushpop(7) 4 >>> h Heap([5, 7, 7]) ''' return heapq.heappushpop(self._queue, item) def replace(self, item): '''Pop one item and push the received one into the heap queue Note that this method is more efficient than calling both pop() and push() separately. >>> h = Heap() >>> h.replace(3) Traceback (most recent call last): ... IndexError: index out of range >>> h.push(7) >>> h.replace(2) 7 >>> h Heap([2]) >>> h.push(3) >>> h Heap([2, 3]) >>> h.replace(1) 2 >>> h.replace(9) 1 >>> h Heap([3, 9]) ''' return heapq.heapreplace(self._queue, item) """ Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

11 14

Python Bytecode Verifier
by Kornél Pál Sept. 20, 2009

Sept. 20, 2009

Hi, I've created a Python Bytecode Verifier in Python. I'm not a Python guru so I borrowed coding patterns from C/C++. I also created this with C portability in mind. The only reason I used Python was to experiment with Python and was easier to morph code during development. If this program were ported to C it would only need 8 bytes per opcode (some additional storage to track blocks) and a single pass. I haven't found backward jumps to previously unused code in any compiled Python code but it can easily be supported. In that case some more partial passes are required. I also was able to successfully verify all Python files in the Python-2.5.2 source package. The verification algorythm should be quite complete but I may have missed some limitations of the interpreter that could be checked by the verifier as well. The ability to create this verfier proved that although Python bytecode is designed for a dynamically typed interpreter, is still easily verifiable. I am willing port this code C but only in the case if there is any chance to be included in Python. I believe that Python in general would benefit having the ability to safely load .pyc files and create code objects on the fly. Both Java and .NET have the ability to safely load compiled byte code. .NET Framework, just like Python also has the ability to create and execute new code at run-time. You may feel that enabling closed source applications and/or creating a multi-language runtime would hurt Python but both of these have contributed to the success of Java (both the language and the runtime). Kornél

2 2

os.listdir with current working directory as default
by Tarek Ziadé Aug. 1, 2009

Aug. 1, 2009

Hello, A very small change: what about making the path argument optional for os.listdir ? and use the current working directory if not provided. listdir(...) listdir(path=None) -> list_of_strings Return a list containing the names of the entries in the directory. path: path of directory to list. If path is None, os.getcwd() is used. The list is in arbitrary order. It does not include the special entries '.' and '..' even if they are present in the directory. Cheers Tarek -- Tarek Ziadé | http://ziade.org

11 18

with statement vs. try...except...finally
by Gerald Britton June 3, 2009

June 3, 2009

I'm wondering if the "with" statement should have exception clauses like the "try" statement, even though this seems to defeat part of the reason for the "with" statement. Currently I have a program segment that opens a file and reads a line, something like this (distilled to its elements for illustration): try: f = open('foo') line = f.readline() f.close() except IOError: line = 'default' So that I get a default value if anything goes awry whilst reading the file. If I write it using a "with" statement, I might have: line = 'default' with open('foo') as f: line = f.readline() Fine so far, but what if I want to be more granular? e.g. with "try...except": try: f = open('foo') except IOError: line = "can't open" try: line = f.readline() except IOError: line = "can't read" try: f.close() except IOError: line = "can't close" I can't see how to replace the above try-triplet with a "with" encapsulation. Or, do I have to wrap the "with" statement in try like this: try: with open('foo') as f: line = f.readline() except IOError: line = 'problem with read or close" -- Gerald Britton

11 13

Re: [Python-ideas] 80 character line width vs. something wider
by average May 29, 2009

May 29, 2009

On Tue, May 26, 2009 at 09:47:05AM -0700, Aahz wrote: > On Tue, May 26, 2009, Zooko Wilcox-O'Hearn wrote: >> >> Personally, I would consider "objective arguments" to be controlled, >> repeatasble, studies with quantitative results. I've seen such studies >> about light-background-dark-foreground vs. dark-foreground- >> light-background, which is why I use the former now. I haven't seen >> such studies about line width, especially not with Python text as >> opposed to English text. > > Personally, I would be amazed to see any significant difference between > the two foreground/background combinations you list. ;-) BTW, I know the topic's pretty dead by now, but there is a pretty conclusive argument on this topic (which I bring back up only because I find myself continually annoyed at most IDE's). If the medium *emits* light, it's significantly better to have a dark background (the reverse being true if the medium is reflective--like the surface of a book or Kindle). Several arguments *support* this and none *contradict* it: 1) Examine the edge cases. Which would you fatigue in front of faster: a monitor emitting and filled with bright white light or one that is dark? (duh) 2) Adjust screen refresh to 60Hz. Text your source code on the screen with the different foregrounds. Stand back and move your head around. With which do you see the flicker? (answer: white background) 3) The retina has to resolve the actual raw and otherwise meaningless dark and light spots into useful information. Partly due to the resolving power of the retina being greater than the display and also because of interpolation effects that happen within the retina itself, there's going to be a slight advantage if the actual data on the page is emitting the light (i.e. a light foreground) rather than having to compete with trying to stand out against the white wash bleeding over. The significance of this problem is inversely proportional to the fatness of the fonts (finer fonts-->more problem). 4) It's easier on your monitor and your ears (if your using a CRT). The photon gun only has to emit streams of particles on the "on" bits and can be off less, saving your shadow mask and those 15kHz screams emitting from the back. 5) Less power consumption. I'd estimate the power savings roughly 60% and higher on CRTs. If anyone has one of those $15 Kill-o-Watt power measuring device they can check. And for those who would otherwise take a populist approach to the problem ("gee if it was better, it would have been adopted by now...."), consider that the main reason that white backgrounds are common is because the guys at XEROX Park took either a naive approach to wanting a display "more like the real world" or because they knew that it would be that much more striking and novel at a time when green monochrome was the norm. It was then subsequently and blindly copied by Apple who wanted to be like XEROX, and then again by MS Windows which wanted to compete with the Mac. I seriously doubt ($0.02 blah blah) that any of these companies propagated the idea based on research into white-foreground effects on neural fatigue and such. And now it's a serious chore to re-customize all the colors to work well with a dark background and that totally *blows*.... Hope that puts THAT issue to rest.... ;^) marcos

4 3

80 character line width vs. something wider
by Aaron Rubin May 27, 2009

May 27, 2009

I realize that this is a religious debate which has gone on for many centuries. I appeal to the scientific aspects, with a distinct avoidance of preference and emotion. Preference might be easily explained by "right brain" vs "left brain" preference, but either way, it is merely a preference and I want to stick to facts. Here is a list I have compiled of facts which support a wider than 80 character line width standard (in Python, specifically). Please add to them, subtract from them, or add to the other side of the debate, but please avoid the usage of the word "readable" (which implies preference), unless you are referring to a scientific study of readability, which would be great. 1) Python is three things which the standard was not designed for: One: Object Oriented. Two: Not Hungarian notation Three: Mandatorily uses *whitespace* as its defintion for code blocks. Let me explain each one in a bit more detail: Object Oriented: Because it is not functional-style programming, but instead OO, you have to give defintion as to what object type you are using before using it. This makes definitions and usage longer than in functional programming (when 80 character widths were invented). PhazeMonkey.Hardware.FrameSource.Abstract.Framegrabber is an example (and not an extreme one) of a class (55 characters already) in a rather large code base. Not Hungarian: Not only is Python not Hungarian (in general), but the PEP-8 specifically tells us to use longer, more descriptive variable names. hasInstrumentControllerPhaseDither is an example. Many variables are 15-20 characters and oftentimes longer. How many of these variables can you fit into a line if we are limited to 80? Whitespace: Python is very unique in that it *uses* whitespace for code blocking. It turns out to be very useful, since it visually cues the reader where code blocks begin and end by mandate. This creates many situations where code *starts* at the 10th indentation (40 characters in our standard, 80 characters in some Python standards). Even in normal "great design" mode (which takes more time again), you can't help it....your code starts at the 6th indentation level often. (28 characters, more than 30% of 80 characters already gone. Now how many variables or class names can you fit?) Whitespace (2): Because Python uses whitespace as its sole method of code blocking and because this is the visual cue for code blocks, wrapping lines obfuscates this and makes the reader think about whether this whitespace was there for a code block, or a line-wrap. Thinking about intention of code slows us down. 2) Many of the libraries that are widely used do not adhere to the 80 character width line standard. wxPython, NumPy and Boa Constructor are a few, but I'm sure there are many, many more. Many libraries do adhere to 80 character line width as well. However, a library which is written in 80 characters still fits the paradigm of those which are wider and therefore backward compliant. In other words, if your tools are geared toward 80 character line widths and you are now looking at a wider width, things become quite difficult. The other way around is fine. 3) Writing new code in 80 character line widths takes more time. If I have to worry about hitting this width, I have to de-concentrate my efforts of writing logical code and concentrate instead on how exactly to format this line of code (where to break it, etc....there are a number of rules attached to wrapping lines of code). Then I have to re-concentrate on the actual task at hand. Alternatively, I can code it up without worrying, then when convenient, take some time to reformat to 80 character width. Either way, more time. 4) Code searching. IDEs have powerful searching features. They list all the lines of a file (or files) which match the string you are searching for. If things are in one line, this search is meaningful and you can read it like you can code. If a line of code actually spans two (or more) lines of code, the search is no longer contextually useful and you have to click on each item to see what's actually going on. This feature is used heavily in many environments (especially large code bases) to save time, but time is either lost finding the actual context of a found string, or the search tool is avoided altogether because it does not provide meaningful results (i.e. a predictive waste of time) 5) Monitors are getting bigger, wider, cheaper. This allows us to have two files, side-by-side on a screen that are *both* 120 character width (or even wider), without changing font sizes. 6) Tools are cheap. Time isn't. Get a second monitor, get a more powerful editor, do whatever it takes to save time. If viewing more information at one time is important, then we should try to make that possible with technology, not time. 7) Python is designed to be written more like English than other programming languages. English is written horizontally, not vertically. In furtherance to an attempt to make "readability" an objective argument, here is a scientific study which finds that greater character width lines improve readability: http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp.<http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp> To summarize, the study found that of the choices of 35, 55, 75 and 95 character lengths, 95 was able to be read the fastest. Please note that they did not try 115, 135, etc. and that they found their maximum data point at the farthest edge of their study. One can conclude that they probably should have gone even further to determine where (if ever) it tapers off. This study focuses on reading English, not on reading code. But given the first sentence of this point, it should at least loosely correlate. At any rate, it's an attempt at something scientific on the issue. Thanks for the time spent reading this long-ish post. Thanks for your feedback if you provide it. - Aaron Aaron Rubin Software Engineering Manager 4D Technology

33 89

pipe function for itertools?
by Donald 'Paddy' McCarthy May 27, 2009

May 27, 2009

Hi, I have a blog entry, http://paddy3118.blogspot.com/2009/05/pipe-fitting-with-python-generators.h…, in which I define a helper function to get around the problem of reversion in the nesting of generators and the proliferation of nested brackets. See the blog entry for a fuller treatment, but in essence if you have a series of generators, and want the data to conceptually flow like this: gen1 -> gen2 -> gen3 -> gen4 ... You have to write: ...(gen4(gen3(gen2(gen1()))))... With pipe, you would write: pipe(gen1, gen2, gen3, gen4, ...) Which you could use like this: for data in pipe(...): do_something_with_data() If I use dots for indentation, then maybe the definition will come through to the group: def pipe(*cmds): ....gen = cmds[0] ....for cmd in cmds[1:]: ....... gen = cmd(gen) ....for x in gen: ....... yield x A couple of readers thought that it might be a good tool for itertools to have. There are other solutions out there that use classes to overload '|', or, but it seems more natural to me to use pipe(), even though I am a heavy Unix user. Please discuss.

5 4

proxy objects and call-stack depend globals
by Ronny Pfannschmidt May 25, 2009

May 25, 2009

Hi, for testing and dealing with thread-depend stdout/stderr redirection (__so__ helpfull for embedded debug consoles in gui apps), i wrote a simple lib to create global proxy objects which may point to different objects depending on the context of the call stack. The whole thing is based on a simple proxy implementation taken from werkzeug and a context managers to handle the current context of that global. I extracted the proxying bits to a base class I'ts availiable at http://pypi.python.org/pypi/pyscope and http://bitbucket.org/RonnyPfannschmidt/pyscope/overview/ Im hoping for some input on how to do it better and if something along that lines would be a usefull extension to the stdlib. Regards Ronny Pfannschmidt

1 0

Re: [Python-ideas] 80 character line width vs. something wider
by Jess Austin May 22, 2009

May 22, 2009

On Thu, May 21, 2009 at 1:01 PM, Aaron Rubin <aaron.rubin(a)4dtechnology.com> wrote: > subjective ones. Arguing over *how* to break lines is actually a pretty > strong argument that time is wasted spent on such issues. A longer line > width would reduce these arguments, since less would need to be wrapped. This is an unsupported, and IMHO largely incorrect, assumption. Several correspondents have noted that they most often overrun their intended line length by one or two characters. Just as there's nothing magical about the number "80", there's nothing magical about "81" or "82" either. In a regime of 90-character lines, the limit will most often be exceeded by one or two characters. The same will happen in a regime of 100-character lines, etc. We'll still need to break lines, and wrapping them in parentheses will still be the best way to do that. This sentiment reminds me of those road-construction enthusiasts who are always certain that the _next_ newly-built highway will end traffic jams forever. What happens instead is that the new road fills with cars too, and it still takes forever to get to work. cheers, Jess ps. ride a bike!

3 4

Re: [Python-ideas] 80 character line width vs. something wider
by Aaron Rubin May 22, 2009

May 22, 2009

On Tue, May 19, 2009 at 10:51 AM, Mike Meyer <mwm(a)mired.org> wrote: > On May 19, 2009, at 12:43, Aaron Rubin <aaron.rubin(a)4dtechnology.com> > wrote: > > I realize that this is a religious debate which has gone on for many > centuries. I appeal to the scientific aspects, with a distinct avoidance of > preference and emotion. Preference might be easily explained by "right > brain" vs "left brain" preference, but either way, it is merely a preference > and I want to stick to facts. Here is a list I have compiled of facts which > support a wider than 80 character line width standard (in Python, > specifically). Please add to them, subtract from them, or add to the other > side of the debate, but please avoid the usage of the word "readable" (which > implies preference), unless you are referring to a scientific study of > readability, which would be great. > > > Most of you points don't stand up under inestigation. Of course, negating > them doesn't support am 80 character limit by itself. > > 1) Python is three things which the standard was not designed for: One: > Object Oriented. Two: Not Hungarian notation Three: Mandatorily uses > *whitespace* as its defintion for code blocks. Let me explain each one in a > bit more detail: Object Oriented: Because it is not functional-style > programming, but instead OO, you have to give defintion as to what object > type you are using before using it. This makes definitions and usage longer > than in functional programming (when 80 character widths were invented). > PhazeMonkey.Hardware.FrameSource.Abstract.Framegrabber is an example (and > not an extreme one) of a class (55 characters already) in a rather large > code base. > > > This type of reference is considered by some to be bad style. See the "Law > of Demeter" for more information. > The Law of Demeter applies to objects referenced second-hand. The class name given is an example of a hierarchy of modules, not one class reaching through a second class to get at the class members it uses. > Not Hungarian: Not only is Python not Hungarian (in general), but the > PEP-8 specifically tells us to use longer, more descriptive variable names. > hasInstrumentControllerPhaseDither is an example. Many variables are 15-20 > characters and oftentimes longer. > > > This appears to be false. A quick check of the standard library finds > between 1 and 2 percent of variable references to have fewer than 15 > characters, rising to 8 percent of unique names. This hardly qualified as > many. > do you mean greater than 15 characters? If not, then I don't see your point. At any rate, 8 percent of unique names seems statistically relevant. 8 percent of how many? If the number is 100,000, then I would say that 8,000 variable names qualifies as "many". > Whitespace: Python is very unique in that it *uses* whitespace for code > blocking. It turns out to be very useful, since it visually cues the reader > where code blocks begin and end by mandate. This creates many situations > where code *starts* at the 10th indentation (40 characters in our standard, > 80 characters in some Python standards). > > > This also appears to be false - the standard library has fewer than 200 > lines out of over 80,000 that start that deep. "Rarely" would seem to be > more accurate than "many". > 200 lines might qualify as many. Regardless, there are quite a number of lines which do. Let's not argue over the meaning of "many". As long as "some" exist, the point remains. > > Even in normal "great design" mode (which takes more time again), you can't > help it....your code starts at the 6th indentation level often. (28 > characters, more than 30% of 80 characters already gone. Now how many > variables or class names can you fit?) > Whitespace (2): Because Python uses whitespace as its sole method of > code blocking and because this is the visual cue for code blocks, wrapping > lines obfuscates this and makes the reader think about whether this > whitespace was there for a code block, or a line-wrap. Thinking about > intention of code slows us down. > > 2) Many of the libraries that are widely used do not adhere to the 80 > character width line standard. wxPython, NumPy and Boa Constructor are a > few, but I'm sure there are many, many more. > > > Which just goes to show that you don't hve to be constrained by the PEP if > you don't want to. Notvthat doing so is a good idea. > > 3) Writing new code in 80 character line widths takes more time. If I > have to worry about hitting this width, I have to de-concentrate my efforts > of writing logical code and concentrate instead on how exactly to format > this line of code (where to break it, etc....there are a number of rules > attached to wrapping lines of code). Then I have to re-concentrate on the > actual task at hand. Alternatively, I can code it up without worrying, then > when convenient, take some time to reformat to 80 character width. Either > way, more time. > > > On the other side, the Oulipo school of writing believes that writing with > apparently arbitrary constraints improves the results. I find that if I'm > running into the 80 character limit with any frequency, it's because my code > is poorly structured and in need of a reworking. > That can definitely be a symptom of bad code. Doesn't mean it's the only reason for it, however. > > 4) Code searching. IDEs have powerful searching features. They list all > the lines of a file (or files) which match the string you are searching for. > If things are in one line, this search is meaningful and you can read it > like you can code. If a line of code actually spans two (or more) lines > of code, the search is no longer contextually useful and you have to click > on each item to see what's actually going on. This feature is used heavily > in many environments (especially large code bases) to save time, but time is > either lost finding the actual context of a found string, or the search tool > is avoided altogether because it does not provide meaningful results (i.e. a > predictive waste of time) > > > In that case, you need a better IDE; it should either show more context, > allow multiline searches, or both. Both of these things will help even if > you never wrap your lines. > > 5) Monitors are getting bigger, wider, cheaper. This allows us to have two > files, side-by-side on a screen that are *both* 120 character width (or > even wider), without changing font sizes. > > > Printers aren't. > > 6) Tools are cheap. Time isn't. Get a second monitor, get a more powerful > editor, do whatever it takes to save time. If viewing more information at > one time is important, then we should try to make that possible with > technology, not time. > > > This point seems to be at best neutral to where or if you wrap lines. A > good IDE will wrap the display and indicate it did so. > > 7) Python is designed to be written more like English than other > programming languages. English is written horizontally, not vertically. In > furtherance to an attempt to make "readability" an objective argument, here > is a scientific study which finds that greater character width lines > improve readability: <http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp> > http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp. > To summarize, the study found that of the choices of 35, 55, 75 and 95 > character lengths, 95 was able to be read the fastest. > > > Please note that the study *started* by pointing out that other studies > existed which found the best line length to be anywhere from 35 to 85 > characters, and gave no reason for trusting their results rather than the > earlier studies. I would claim that all studies that looked at written > languages - as opposed to programs - were inapplicable. They almost > certainly used variable width fonts and a left justification (if not both), > either of which has more effect on readability than line length. > > Thanks for the time spent reading this long-ish post. Thanks for your > feedback if you provide it. > > > You really need better justification - studies of program comprehension, > not English reading speed, code bases that support your claims about written > code, etc. > I would love to see these studies done, but at this time I cannot find them. The closest I could come (and I disclaimed that it was only a loose connection) was the study I referenced. > > <mike >

9 9