[Tutor] Can someone help me to improve this segment?

[Danny Yoo]

Hi Titu, welcome to Tutor!

On Mon, 29 Oct 2001, Titu Kim wrote:

> I am little obssessed with my own coded after i found out it is slow.

That's ok; Optimizing programs is a good goal as long as you know where to
pinpoint your efforts.


> I am trying to construct a html table string from a result that i
> obtain from database.My code segment look like this: assume result
>  holds 2D list.
>
>  *******  method********************************
>  def f1(item):
>      return "<td>"+str(item)+"</td>"
>  *****************************************
>  ************main program piece***********
>  tab = "<table align='center' border='1'>\n"
>  for x in result:
>     tab = tab + "<tr>"
>     tab = tab + string.join(map(f1,x),"")+"</tr>\n"
>  tab = tab + "</table>"


I think that the string concatenating may be the culprit: it's somewhat
expensive to do a lot of small string concatentations.  For example:

###
>>> numbers = map(str, range(100000))
>>> mystr = ""
>>> for element in numbers:
...     mystr = mystr + str(element)
... 
###

causes my computer to grind to a halt: Python spends a lot of time trying
to put things together, and it just fumbles.  Imagine someone writing a
word on a chalkboard, erasing it, writing it again, erasing it, writing it
again... that's a handwavy way of seeing that doing concatenation like
this is expensive.


What Python's better at is taking a list and joining its elements into a
string:

###
>>> mystr = string.join(numbers, "")
>>> len(mystr)
488890
###

It's better at it because it expects not to do the "write/erase/write"
sort of thing: it really does try to writing things out all at once.  And
it does it pretty quickly.



Your code:

>  tab = "<table align='center' border='1'>\n"
>  for x in result:
>     tab = tab + "<tr>"
>     tab = tab + string.join(map(f1,x),"")+"</tr>\n"
>  tab = tab + "</table>"

can use a similar trick if you use a list to hold all the temporary string
fragments, and this is probably one of the best ways to speed it up.



Alternatively, if you like working with files, you can use a StringIO()
object to temporarily hold things while you're concatenating:

###
>>> import StringIO
>>> stringout = StringIO.StringIO()
>>> for element in numbers:
...     stringout.write(element)
...
>>> len(stringout.getvalue())
488890 
###

Subjectively, using StringIO does feel a little slower than using the
string.join()ing, but it also works effectively.



Hope this helps!