Mailman 3 Re: [Python-ideas] Create a StringBuilder class and use it everywhere - Python-ideas

newer
Re: [Python-ideas] IP addressing...

Re: [Python-ideas] Create a StringBuilder class and use it everywhere

older
Add from __experimental__ import...

k_bx

25 Aug 2011 25 Aug '11

8:58 p.m.

This doesn't seem nicer to read and write to me than the list form. I also do not see any reason to believe it will stop people from doing it the quadratic way if the ubiquitous make-a-list-then-join idiom does not.

The whole point is that people don't write u''.join idiom just because they don't know it is slow. And when they see StringBuilder -- they can ask themselves "why is he using that". I don't mind using u''.join, but it just doesn't make people think about speed at all. The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow". With StringBuilder used widely that would not be the case.

Show replies by thread

ron3200

25 Aug 25 Aug

9:54 p.m.

New subject: Create a StringBuilder class and use it everywhere

On Thu, 2011-08-25 at 18:28 +0300, k_bx wrote:

...

The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

I think a FAQ on "How can I make my python program faster?", with suggestions such as using list .join for building large strings instead of using += would be better. There probably already is one some place... Yep... http://wiki.python.org/moin/PythonSpeed/PerformanceTips This in my opinion is more about fitting the code to the problem than it is about speeding up general python code. I once wrote a text comparison engine that solved cryptograms by comparing to a text source. A large text source was read into a dictionary of words to be compared to. At first it was quite slow, but by presorting the data and putting it into smaller dictionaries, it sped up the program by several order of magnitudes. Cheers, Ron

Antoine Pitrou

10:10 p.m.

New subject: Create a StringBuilder class and use it everywhere

On Thu, 25 Aug 2011 18:28:34 +0300 k_bx wrote:

...

I don't mind using u''.join, but it just doesn't make people think about speed at all.

Realistically, not many workloads have performance issues with string concatenation in the first place. So not caring is the right thing to do in most cases.

...

The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Different implementations having different performance characteristics is not totally unexpected, is it? (and I'm sure the PyPy developers wouldn't mind adding another hack) Regards Antoine.

Masklinn

10:20 p.m.

New subject: Create a StringBuilder class and use it everywhere

On 2011-08-25, at 18:40 , Antoine Pitrou wrote:

...

...
The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Different implementations having different performance characteristics is not totally unexpected, is it? (and I'm sure the PyPy developers wouldn't mind adding another hack) This one can not be done, as it relies on knowing there's only one reference to the string (so it can be realloc'd in place), therefore on using a refcounting GC.

Since Pypy does not use refcounting, it can't do that as a rule (it might be possible to handle it for a limited number of cases via escape analysis, proving there can be only one reference to the string, but I'd say there are more interesting things to use escape analysis for). Also, http://twitter.com/#!/alex_gaynor/status/104326041920749569

...

Wish CPython didn't contains hacks which make str += str faster, sometimes, depending on refcounting details :(

Stefan Behnel

10:28 p.m.

New subject: Create a StringBuilder class and use it everywhere

k_bx, 25.08.2011 17:28:

...

I don't mind using u''.join, but it just doesn't make people think about speed at all.

When I see something like a StringBuilder, I guess the first thing I'd wonder about is why the programmer didn't just use StringIO() or even just ''.join(). That makes the code appear much more magic than it eventually turns out to be when looking closer. Plus, it's doomed to be slower, simply because it goes through more indirections. You may be right that using StringIO won't make people think about speed. Somebody who doesn't know it would likely go: "oh, that's nice - writing to a string as if it were a file - I get that". So it tells you what it does, instead of distracting you into thinking about performance implications. That's the beauty of it. Optimisations are just that: optimisations. They are orthogonal to what the code does - or at least they should be. Even string concatenation can preform just fine in many cases.

...

The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Sounds like yet another reason not to do it then. Seriously, there are hardly any language runtimes out there where continued string concatenation is efficient, let alone guaranteed to be so. You just shouldn't expect that it is. The optimisation in CPython was simply done because it *can* be done, so that simple cases (and stupid benchmarks) can continue to use simple concatenation and still be efficient. Well, in some cases at least. Stefan

Antoine Pitrou

10:28 p.m.

New subject: Create a StringBuilder class and use it everywhere

Le jeudi 25 août 2011 à 18:50 +0200, Masklinn a écrit :

...

On 2011-08-25, at 18:40 , Antoine Pitrou wrote:

...
...
The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Different implementations having different performance characteristics is not totally unexpected, is it? (and I'm sure the PyPy developers wouldn't mind adding another hack) This one can not be done, as it relies on knowing there's only one reference to the string (so it can be realloc'd in place), therefore on using a refcounting GC.

Ah, you're right. However, PyPy has another (and quite broader) set of optimizations available: http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#strin... Besides:

...

Since Pypy does not use refcounting, it can't do that as a rule (it might be possible to handle it for a limited number of cases via escape analysis, proving there can be only one reference to the string, but I'd say there are more interesting things to use escape analysis for).

The CPython optimization itself works in a limited number of cases, because having a refcount of 1 essentially means it's a local variable, and because of the result having to be stored back immediately in the same local variable (otherwise you can't recycle the original object's storage). Regards Antoine.

Stefan Behnel

10:51 p.m.

New subject: Create a StringBuilder class and use it everywhere

Antoine Pitrou, 25.08.2011 18:58:

...

Le jeudi 25 août 2011 à 18:50 +0200, Masklinn a écrit :

...
On 2011-08-25, at 18:40 , Antoine Pitrou wrote:

...
...
The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Different implementations having different performance characteristics is not totally unexpected, is it? (and I'm sure the PyPy developers wouldn't mind adding another hack) This one can not be done, as it relies on knowing there's only one reference to the string (so it can be realloc'd in place), therefore on using a refcounting GC.

Ah, you're right. However, PyPy has another (and quite broader) set of optimizations available: http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#strin...

And its JIT could potentially just enable its string-join optimisation automatically when it sees that a variable holds a string, and is never being assigned to inside of a loop or sequence of operations except for the += operator. Any other operation on the string would then just turn it back into a normal string by joining it first. But this is seriously getting off-topic now. Stefan

Masklinn

11:05 p.m.

New subject: Create a StringBuilder class and use it everywhere

On 2011-08-25, at 18:58 , Antoine Pitrou wrote:

...

Le jeudi 25 août 2011 à 18:50 +0200, Masklinn a écrit :

...
On 2011-08-25, at 18:40 , Antoine Pitrou wrote:

...
...
The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

Different implementations having different performance characteristics is not totally unexpected, is it? (and I'm sure the PyPy developers wouldn't mind adding another hack) This one can not be done, as it relies on knowing there's only one reference to the string (so it can be realloc'd in place), therefore on using a refcounting GC. Ah, you're right. However, PyPy has another (and quite broader) set of optimizations available: http://codespeak.net/pypy/dist/pypy/doc/interpreter-optimizations.html#strin... Yeah but none of them has reached enough utility to be the default pypy state, so it's not like it's a good idea to rely on them (it's nice to know these options exist as they can be nice in a per-case basis though).

...

Besides:

...
Since Pypy does not use refcounting, it can't do that as a rule (it might be possible to handle it for a limited number of cases via escape analysis, proving there can be only one reference to the string, but I'd say there are more interesting things to use escape analysis for).

The CPython optimization itself works in a limited number of cases, because having a refcount of 1 essentially means it's a local variable, and because of the result having to be stored back immediately in the same local variable (otherwise you can't recycle the original object's storage). True, I realized afterwards I should probably have written "a likely even more limited number of cases than CPython". Oh well.

Greg Ewing

26 Aug 26 Aug

5:57 a.m.

New subject: Create a StringBuilder class and use it everywhere

k_bx wrote:

...

The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

To me that suggests it may have been a mistake to try to optimise += at all in CPython, as it gives people misleading expectations. -- Greg

Amaury Forgeot d'Arc

12:44 p.m.

New subject: Create a StringBuilder class and use it everywhere

2011/8/26 Greg Ewing

...

The most popular (as from what I can see) thing right now where people

...
start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow".

To me that suggests it may have been a mistake to try to optimise += at all in CPython, as it gives people misleading expectations.

The author of this optimization is also the lead designer of PyPy... -- Amaury Forgeot d'Arc

Spectral One

30 Aug 30 Aug

9:16 p.m.

New subject: Expanding statistical functions in Python's std. lib.

Wandering about, looking up statistics info for a program I was writing, I found a recommendation to add various useful 'special functions' to C's math library: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1069.pdf The arguments in that paper make a lot of sense to me, and apply well to Python. They came up with a good list, IMnsHO. I'd recommend implementing this list in some form as library functions in Python. Blindly copying wouldn't end up particularly 'Pythonic;' tweaking the API is required. Some of the selection choices, such as returning real only, ought to be reevaluated, for example. Obviously, any of the decisions to keep things C-like rather than object-oriented ought to shift, as well. Function names are only important as far as they are clear. I suggest naming per <general category><specific case> e.g. distribution_t(), or dist_F(), and include modification for algebraic order, as well, so gamma() and log_gamma(). That said, anything clear is fine. Thoughts on the matter? I noticed that the math library in 2.7+ added the gamma and log(gamma) functions, already, which was nice. Obviously, most, if not all, are already present in extensions modules such as NumPy, but there is value in having these things built into the language. "Batteries included, "and all that. By the by, if that is far too much for one suggestion, then please just treat this as a suggestion to add just the incomplete beta function. (P-values for binomial, F, and t are all nice, too, though with inc. beta, they aren't terrible to generate. I really think they should be included in the standard library.) -Nate

Guido van Rossum

1 Sep 1 Sep

12:35 a.m.

New subject: Expanding statistical functions in Python's std. lib.

You didn't get any responses AFAICT. That doesn't mean nobody is interested -- perhaps your proposal is simply too general? Do you feel up to making some more specific recommendations about the exact list of functions to add? It's easier to criticize a concrete proposal. Do you feel up to producing a patch that just adds the incomplete beta function? --Guido On Tue, Aug 30, 2011 at 8:46 AM, Spectral One wrote:

...

Wandering about, looking up statistics info for a program I was writing, I found a recommendation to add various useful 'special functions' to C's math library: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1069.pdf

The arguments in that paper make a lot of sense to me, and apply well to Python. They came up with a good list, IMnsHO. I'd recommend implementing this list in some form as library functions in Python.

Blindly copying wouldn't end up particularly 'Pythonic;' tweaking the API is required. Some of the selection choices, such as returning real only, ought to be reevaluated, for example. Obviously, any of the decisions to keep things C-like rather than object-oriented ought to shift, as well.

Function names are only important as far as they are clear. I suggest naming per <general category><specific case> e.g. distribution_t(), or dist_F(), and include modification for algebraic order, as well, so gamma() and log_gamma(). That said, anything clear is fine.

Thoughts on the matter? I noticed that the math library in 2.7+ added the gamma and log(gamma) functions, already, which was nice. Obviously, most, if not all, are already present in extensions modules such as NumPy, but there is value in having these things built into the language. "Batteries included, "and all that.

By the by, if that is far too much for one suggestion, then please just treat this as a suggestion to add just the incomplete beta function. (P-values for binomial, F, and t are all nice, too, though with inc. beta, they aren't terrible to generate. I really think they should be included in the standard library.)

-Nate _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

Robert Kern

3:49 a.m.

New subject: Expanding statistical functions in Python's std. lib.

On 8/31/11 2:05 PM, Guido van Rossum wrote:

...

You didn't get any responses AFAICT. That doesn't mean nobody is interested -- perhaps your proposal is simply too general? Do you feel up to making some more specific recommendations about the exact list of functions to add? It's easier to criticize a concrete proposal. Do you feel up to producing a patch that just adds the incomplete beta function?

It shows up deeply mis-threaded under "Create a StringBuilder class and use it everywhere" in my client. Perhaps Spectral One should try reposting it so that it shows up as a new thread. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Guido van Rossum

3:53 a.m.

New subject: Expanding statistical functions in Python's std. lib.

On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote:

...

On 8/31/11 2:05 PM, Guido van Rossum wrote:

...
You didn't get any responses AFAICT. That doesn't mean nobody is interested -- perhaps your proposal is simply too general? Do you feel up to making some more specific recommendations about the exact list of functions to add? It's easier to criticize a concrete proposal. Do you feel up to producing a patch that just adds the incomplete beta function?

It shows up deeply mis-threaded under "Create a StringBuilder class and use it everywhere" in my client. Perhaps Spectral One should try reposting it so that it shows up as a new thread.

What client is that? In my GMail (for once) it shows up as a new thread with subject "Re: [Python-ideas] Expanding statistical functions in Python's std. lib." I guess your client got confused by some of these headers: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> In-Reply-To: <4E56E859.3090504@canterbury.ac.nz> -- --Guido van Rossum (python.org/~guido)

Robert Kern

4:34 a.m.

New subject: Expanding statistical functions in Python's std. lib.

On 8/31/11 5:23 PM, Guido van Rossum wrote:

...

On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote:

...
On 8/31/11 2:05 PM, Guido van Rossum wrote:

...
You didn't get any responses AFAICT. That doesn't mean nobody is interested -- perhaps your proposal is simply too general? Do you feel up to making some more specific recommendations about the exact list of functions to add? It's easier to criticize a concrete proposal. Do you feel up to producing a patch that just adds the incomplete beta function?

It shows up deeply mis-threaded under "Create a StringBuilder class and use it everywhere" in my client. Perhaps Spectral One should try reposting it so that it shows up as a new thread.

What client is that?

Thunderbird, via GMane, which may or may not be adding more confusion to the mix.

...

In my GMail (for once) it shows up as a new thread with subject "Re: [Python-ideas] Expanding statistical functions in Python's std. lib." I guess your client got confused by some of these headers:

References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> In-Reply-To: <4E56E859.3090504@canterbury.ac.nz>

And indeed, it shows up threaded under Greg Ewing's Aug 25 post to the StringBuilder thread. Email threading is something of an art, but I'm not sure it's right to say that my client is getting "confused" by taking the In-Reply-To header at its word. ;-) Anyways, that's why I suspect he's not getting many responses. As to the substance of the proposal, I'm -0 on having the full complement of statistical distribution functions and +0 on adding just the incomplete beta function. Personally, I will never use any of them since I can get them from scipy. I am at least going to be using numpy to generate any of the test statistics that I would pass through these functions. I don't see anything particularly compelling about having them in the math module as opposed to a third party module (be it scipy or something lighter-weight). That said, having a good complement of common special functions that can be used to build up a variety of less-common functions is a good thing to have in a standard library. I think you could defend adding the incomplete beta function on that principle, if nothing else. You could make a similar argument for the Bessel functions j0(), j1() and jn(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

David Townshend

10:09 a.m.

New subject: Expanding statistical functions in Python's std. lib.

...

On 8/31/11 5:23 PM, Guido van Rossum wrote:

...
On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote:

...
On 8/31/11 2:05 PM, Guido van Rossum wrote:

...
You didn't get any responses AFAICT. That doesn't mean nobody is interested -- perhaps your proposal is simply too general? Do you feel up to making some more specific recommendations about the exact list of functions to add? It's easier to criticize a concrete proposal. Do you feel up to producing a patch that just adds the incomplete beta function?

It shows up deeply mis-threaded under "Create a StringBuilder class and use it everywhere" in my client. Perhaps Spectral One should try reposting it so that it shows up as a new thread.

What client is that?

Thunderbird, via GMane, which may or may not be adding more confusion to

This had probably been thought about before, but why not include numpy in stdlib? On Sep 1, 2011 1:05 AM, "Robert Kern" wrote: the mix.

...

...
In my GMail (for once) it shows up as a new thread with subject "Re: [Python-ideas] Expanding statistical functions in Python's std. lib." I guess your client got confused by some of these headers:

References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> In-Reply-To: <4E56E859.3090504@canterbury.ac.nz>

And indeed, it shows up threaded under Greg Ewing's Aug 25 post to the StringBuilder thread. Email threading is something of an art, but I'm not

sure

...

it's right to say that my client is getting "confused" by taking the In-Reply-To header at its word. ;-)

Anyways, that's why I suspect he's not getting many responses. As to the substance of the proposal, I'm -0 on having the full complement of statistical distribution functions and +0 on adding just the incomplete beta function.

...

Personally, I will never use any of them since I can get them from scipy. I am at least going to be using numpy to generate any of the test statistics that I would pass through these functions. I don't see anything particularly compelling about having them in the math module as opposed to a third party module (be it scipy or something lighter-weight).

That said, having a good complement of common special functions that can be used to build up a variety of less-common functions is a good thing to have in a standard library. I think you could defend adding the incomplete beta function on that principle, if nothing else. You could make a similar argument for the Bessel functions j0(), j1() and jn().

-- Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Robert Kern

11:02 a.m.

New subject: Expanding statistical functions in Python's std. lib.

On 8/31/11 11:39 PM, David Townshend wrote:

...

This had probably been thought about before, but why not include numpy in stdlib?

Well, it's not particularly germane to this thread since most of the requested functions exist in scipy but not numpy. In any case, numpy is too large, too C, and too actively developed to be a part of the standard library at this time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Christopher King

2 Sep 2 Sep

1:50 a.m.

New subject: Expanding statistical functions in Python's std. lib.

I think this is very Pythonic. On blogs describing python, one of the features they list is batteries included. Statistical functions would be great. I give it +1.

Mike Graham

2:26 a.m.

New subject: Expanding statistical functions in Python's std. lib.

On Tue, Aug 30, 2011 at 11:46 AM, Spectral One wrote:

...

Wandering about, looking up statistics info for a program I was writing, I found a recommendation to add various useful 'special functions' to C's math library: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1069.pdf

The arguments in that paper make a lot of sense to me, and apply well to Python. They came up with a good list, IMnsHO. I'd recommend implementing this list in some form as library functions in Python.

Blindly copying wouldn't end up particularly 'Pythonic;' tweaking the API is required. Some of the selection choices, such as returning real only, ought to be reevaluated, for example. Obviously, any of the decisions to keep things C-like rather than object-oriented ought to shift, as well.

Function names are only important as far as they are clear. I suggest naming per <general category><specific case> e.g. distribution_t(), or dist_F(), and include modification for algebraic order, as well, so gamma() and log_gamma(). That said, anything clear is fine.

Thoughts on the matter? I noticed that the math library in 2.7+ added the gamma and log(gamma) functions, already, which was nice. Obviously, most, if not all, are already present in extensions modules such as NumPy, but there is value in having these things built into the language. "Batteries included, "and all that.

By the by, if that is far too much for one suggestion, then please just treat this as a suggestion to add just the incomplete beta function. (P-values for binomial, F, and t are all nice, too, though with inc. beta, they aren't terrible to generate. I really think they should be included in the standard library.)

I'm not sure that many people who could make tons of use from statistical functions don't already have cause to be using numpy/scipy. I would certainly be unfortunate if having a little more statistics functionality in the stdlib discouraged people who should be using numpy from doing so. "Batteries included" has always been a bit of an oversell, and as a Python user I don't have any expectation of being able to do fairly-specialized work without third-party modules, nor do I think it's necessarily a net gain for Python if I could. -0 Mike

Ethan Furman

6:20 a.m.

New subject: Expanding statistical functions in Python's std. lib.

Mike Graham wrote:

...

"Batteries included" has always been a bit of an oversell, and as a Python user I don't have any expectation of being able to do fairly-specialized work without third-party modules, nor do I think it's necessarily a net gain for Python if I could.

Well, it is "batteries" not "nuclear reactors" that are included. One must keep these things in perspective. ~Ethan~

4617

Age (days ago)

4625

Last active (days ago)

List overview

Download

19 comments

14 participants

participants (14)

Amaury Forgeot d'Arc
Antoine Pitrou
Christopher King
David Townshend
Ethan Furman
Greg Ewing
Guido van Rossum
k_bx
Masklinn
Mike Graham
Robert Kern
ron3200
Spectral One
Stefan Behnel

Re: [Python-ideas] Create a StringBuilder class and use it everywhere

k_bx

Spectral One

David Townshend

Christopher King

tags

participants (14)