Hi, I am little obssessed with my own coded after i found out it is slow. I am trying to construct a html table string from a result that i obtain from database.My code segment look like this: assume result holds 2D list. ******* method******************************** def f1(item): return "<td>"+str(item)+"</td>" ***************************************** ************main program piece*********** tab = "<table align='center' border='1'>\n" for x in result: tab = tab + "<tr>" tab = tab + string.join(map(f1,x),"")+"</tr>\n" tab = tab + "</table>" print "Content-type:text/html\n" print tab ********************************************** Can anyone improve this code segment? Thanks alot Regards, Kim Titu __________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com
Titu Kim writes:
Hi, I am little obssessed with my own coded after i found out it is slow. I am trying to construct a html table string from a result that i obtain from database.My code segment look like this: assume result holds 2D list. ******* method******************************** def f1(item): return "<td>"+str(item)+"</td>" ***************************************** ************main program piece*********** tab = "<table align='center' border='1'>\n" for x in result: tab = tab + "<tr>" tab = tab + string.join(map(f1,x),"")+"</tr>\n" tab = tab + "</table>" print "Content-type:text/html\n" print tab ********************************************** Can anyone improve this code segment? Thanks alot
If all you do with tab is print it out, you don't need to do all of those string operations, or function calls: print "Content-type: text/html" print "<table align='center' border='1'>" for x in result: print "<tr>" for item in x: print "<td>"+str(item)+"</td>" print "</tr>" print "</table>" Printing output right away would have lower overhead and use less memory. If you know something about the data type of "item", you could also use the % operator instead of the str function. On a 500 by 500 array created via result = [] for x in range(500): result.append(range(500)) I timed my version at 3 seconds, and your version at 12 seconds (both including the array setup time). On a 1000 by 1000 array, my version took 12 seconds and your version took about 95 seconds. -- Seth David Schoen <schoen@loyalty.org> | Its really terrible when FBI arrested Temp. http://www.loyalty.org/~schoen/ | hacker, who visited USA with peacefull down: http://www.loyalty.org/ (CAF) | mission -- to share his knowledge with http://www.freesklyarov.org/ | american nation. (Ilya V. Vasilyev)
Seth David Schoen writes:
If all you do with tab is print it out, you don't need to do all of those string operations, or function calls:
print "Content-type: text/html" print "<table align='center' border='1'>" for x in result: print "<tr>" for item in x: print "<td>"+str(item)+"</td>" print "</tr>" print "</table>"
Printing output right away would have lower overhead and use less memory.
This can be a big deal for efficiency in a very high level language. Building large or complex data structures could have very high overhead; for example, dynamic allocation of space for strings can be a real problem if you do it over and over again hundreds of thousands of times! Anyway, this also has me wondering about the relative speed of different Python iteration idioms (for, while, map, recursion, and now list comprehensions). They're not all applicable to the same problems, but where there is overlap, which are faster than others? -- Seth David Schoen <schoen@loyalty.org> | Its really terrible when FBI arrested Temp. http://www.loyalty.org/~schoen/ | hacker, who visited USA with peacefull down: http://www.loyalty.org/ (CAF) | mission -- to share his knowledge with http://www.freesklyarov.org/ | american nation. (Ilya V. Vasilyev)
I am also looking for the result on this question. --- Seth David Schoen <schoen@loyalty.org> wrote:
Seth David Schoen writes:
If all you do with tab is print it out, you don't need to do all of those string operations, or function calls:
print "Content-type: text/html" print "<table align='center' border='1'>" for x in result: print "<tr>" for item in x: print "<td>"+str(item)+"</td>" print "</tr>" print "</table>"
Printing output right away would have lower overhead and use less memory.
This can be a big deal for efficiency in a very high level language. Building large or complex data structures could have very high overhead; for example, dynamic allocation of space for strings can be a real problem if you do it over and over again hundreds of thousands of times!
Anyway, this also has me wondering about the relative speed of different Python iteration idioms (for, while, map, recursion, and now list comprehensions). They're not all applicable to the same problems, but where there is overlap, which are faster than others?
-- Seth David Schoen <schoen@loyalty.org> | Its really terrible when FBI arrested Temp. http://www.loyalty.org/~schoen/ | hacker, who visited USA with peacefull down: http://www.loyalty.org/ (CAF) | mission -- to share his knowledge with http://www.freesklyarov.org/ | american nation. (Ilya V. Vasilyev)
_______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig
__________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com
Titu Kim writes:
I am also looking for the result on this question. --- Seth David Schoen <schoen@loyalty.org> wrote:
Anyway, this also has me wondering about the relative speed of different Python iteration idioms (for, while, map, recursion, and now list comprehensions). They're not all applicable to the same problems, but where there is overlap, which are faster than others?
In a trivial test, I noticed that the standard way to do an iteration over an index, for i in range(n): pass raises a MemoryError for sufficiently large n (on Unix with a ulimit in effect, limiting how much memory any process can allocate). On the other hand, i = 0 while i < n: pass obviously has no such memory limitation. The memory problems arise from trying to construct the enormous list -- I can reproduce them on this particular system merely by writing a = range(10**7) Now, one thought is that you could simulate the behavior of list data types by creating a class to represent "sparse lists" or the more general "patterned lists", lists in which the nth item is given by a function of n (instead of being calculated and stored beforehand). The advantage here would be that you could say something like r = PatternedList(lambda x: x, 0, 10**7) for i in r: pass The PatternedList class is easy to implement _except_ that I don't think a user-defined object -- other than a subclass of UserList -- can be used in a "for" loop. UserList doesn't seem to help here, because it doesn't actually allow you to override the element lookup behavior of the list, to change the way the list is represented in memory. Anyway, the general point was about resource requirements for different iteration idioms. Recursion requires memory, iteration over a sequence requires memory if you have to build the sequence just for that purpose, and iteration with while _doesn't_ require more than a constant amount of memory (well, maybe if you get up into long integers). If we rewrote code to use a while instead of a "for i in range", our code's memory profile could be much different. -- Seth David Schoen <schoen@loyalty.org> | Its really terrible when FBI arrested Temp. http://www.loyalty.org/~schoen/ | hacker, who visited USA with peacefull down: http://www.loyalty.org/ (CAF) | mission -- to share his knowledge with http://www.freesklyarov.org/ | american nation. (Ilya V. Vasilyev)
Anyway, the general point was about resource requirements for different iteration idioms. Recursion requires memory, iteration over a sequence requires memory if you have to build the sequence just for that purpose, and iteration with while _doesn't_ require more than a constant amount of memory (well, maybe if you get up into long integers). If we rewrote code to use a while instead of a "for i in range", our code's memory profile could be much different.
Seth makes good points about range() pre-allocating potentially large chunks of memory. Historically, xrange() was provided to allow a "lazy evaluation" approach -- instead of pre-allocating, it would do more like Seth suggests, and compute the next "list" element with each iteration. Since Python 2.2, we have the yield keyword, which, when used in a function definition, makes that function a "generator". To 'yield' a value is akin to using a print statement, except the argument is returned in a usable form (ala 'return') -- which ties this thread to an earlier one re 'return' vs. 'print' semantics. When a function encounters a 'yield', execution stops and the calling function gets the yielded falue. When the function is next triggered using .next(), or is iterated over as an object, as in 'for i in myfunction()', then execution picks up right after the 'yield', as if "nothing had happened" (i.e. there's no more disruption to local variables than if a 'print' had been used -- the function behaves as if that's what had happened). So xrange() is being phased out, given 'yield' provides a much more generic approach to lazy evaluation (means a 'just in time' approach to value generation, where you get a next value just when you need it, without a lot of pre-allocation of values -- which you might never get to, if the loop has a break in it, for example). To make this more concrete, here's a lazyrange() function which behaves somewhat like range (lots of improvements and variations possible):
from __future__ import generators # make feature available
def lazyrange(initial,final,inc): while 1: # infinite loop if inc < 0: # break condition if initial <= final: return else: # alternate break condition if initial >= final: return yield initial initial += inc # increment using inc
for i in lazyrange(10,1,-1): print i,
10 9 8 7 6 5 4 3 2 You see above that by containing a yield, lazyrange becomes a generator, and therefore a kind of iterable object (something you can iterate over using 'for'). Note this alternative syntax:
f = lazyrange(1,10,1) # instantiate function object
type(f) # note f's type <type 'generator'>
dir(f) # note builtin generator method next() ['__class__', '__delattr__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']
for i in f: print i,
1 2 3 4 5 6 7 8 9 Kirby
The PatternedList class is easy to implement _except_ that I don't think a user-defined object -- other than a subclass of UserList -- can be used in a "for" loop. UserList doesn't seem to help here, because it doesn't actually allow you to override the element lookup behavior of the list, to change the way the list is represented in memory.
Any ordered sequence type can be used in a "for" loop. Built-in types which qualify include list, tuple, and string. There are lots of additional sequence types in the libraries, such as UserList (no longer needed as of ver. 2.2 since you can subclass built-in types directly). Generally speaking, any class or type which implements __getitem__(self, key) with a numeric key should work in a "for" loop. -- Dethe Elza (delza@burningtiger.com) Chief Mad Scientist Burning Tiger Technologies (http://burningtiger.com) Living Code Weblog (http://livingcode.ca)
participants (4)
-
Dethe Elza -
Kirby Urner -
Seth David Schoen -
Titu Kim