Mailman 3 August 2011 - Python-ideas

Create a StringBuilder class and use it everywhere
by k_bx Aug. 29, 2011

Aug. 29, 2011

Hi! There's a certain problem right now in python that when people need to build string from pieces they really often do something like this:: def main_pure(): b = u"initial value" for i in xrange(30000): b += u"more data" return b The bad thing about it is that new string is created every time you do +=, so it performs bad on CPython (and horrible on PyPy). If people would use, for example, list of strings it would be much better (performance):: … [View More]def main_list_append(): b = [u"initial value"] for i in xrange(3000000): b.append(u"more data") return u"".join(b) The results are:: kost@kost-laptop:~/tmp$ time python string_bucket_pure.py real 0m7.194s user 0m3.590s sys 0m3.580s kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m0.417s user 0m0.330s sys 0m0.080s Fantastic, isn't it? Also, now let's forget about speed and think about semantics a little: your task is: "build a string from it's pieces", or in other words "build a string from list of pieces", so from this point of view you can say that using [] and u"".join is better in semantic way. Java has it's StringBuilder class for a long time (I'm not really into java, I've just been told about that), and what I think is that python should have it's own StringBuilder:: class StringBuilder(object): """Use it instead of doing += for building unicode strings from pieces""" def __init__(self, val=u""): self.val = val self.appended = [] def __iadd__(self, other): self.appended.append(other) return self def __unicode__(self): self.val = u"".join((self.val, u"".join(self.appended))) self.appended = [] return self.val Why StringBuilder class, not just use [] + u''.join ? Well, I have two reasons for that: 1. It has caching 2. You can document it, because when programmer looks at [] + u"" method he doesn't see _WHY_ is it done so, while when he sees StringBuilder class he can go ahead and read it's help(). Performance of StringBuilder is ok compared to [] + u"" (I've increased number of += from 30000 to 30000000): def main_bucket(): b = StringBuilder(u"initial value ") for i in xrange(30000000): b += u"more data" return unicode(b) For CPython:: kost@kost-laptop:~/tmp$ time python string_bucket_bucket.py real 0m12.944s user 0m11.670s sys 0m1.260s kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m3.540s user 0m2.830s sys 0m0.690s For PyPy 1.6:: (pypy)kost@kost-laptop:~/tmp$ time python string_bucket_bucket.py real 0m18.593s user 0m12.930s sys 0m5.600s (pypy)kost@kost-laptop:~/tmp$ time python string_bucket_append.py real 0m16.214s user 0m11.750s sys 0m4.280s Of course, C implementation could be done to make things faster for CPython, I guess, but really, in comparision to += method it doesn't matter now. It's done to be explicit. p.s.: also, why not use cStringIO? 1. it's not semantically right to create file-like string just to join multiple string pieces into one. 2. if you talk about using it in your code right away -- you can see that noone still uses it because people want += (while with StringBuilder you give them +=). 3. it's somehow slow on pypy right now :-) Thanks. [View Less]

15 28

Re: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?]
by David Townshend Aug. 28, 2011

Aug. 28, 2011

I don't know if the intention is to use __experimental__ only for additions to stdlib or whether it would include language changes, but if so, maybe its worth pointing out that language extensions wouldn't be so easy (if even possible) to write as 3rd party on pypi. On Aug 28, 2011 7:10 PM, "Guido van Rossum" <guido(a)python.org> wrote:

1 0

ctypes and extensions
by Jeremy Sanders Aug. 28, 2011

Aug. 28, 2011

I was following the discussion on python-dev about ctypes not being suitable for the standard library for linking to C libraries, and extension modules not suitable for alternative implementations. Ctypes is fragile when it comes linking to libraries whose API might change giving no kind of error. Extension modules are too closely linked to cpython and reference counting. Why not have some sort of intermediate C layer to produce a library which can be immediatedly loaded by any python … [View More]

3 3

Summer of Code Project Idea: Python Apps in the Browser
by Raymond Hettinger Aug. 26, 2011

Aug. 26, 2011

The http://gumbyapp.com/ project succeeded in getting Python to run in the browser but it can't import pure python modules (because there is no filesystem in the browser). I think it would be wonderful to beef-up this project by bundling-in the rest of the standard library. Gumby was built by compiling CPython to LLVM and then generating Javascript. ISTM it would be possible to write a script to transform pure python standard library modules into C strings that could also be part of the … [View More]

6 7

Re: [Python-ideas] Create a StringBuilder class and use it everywhere
by k_bx Aug. 26, 2011

Aug. 26, 2011

Ok, so while I think that cStringIO.StringIO is not what we shoud use, io.StringIO should be okay (since it wasn't explpicitly created to be file-like), so there's no reason not to use that (and it performs good, on pypy also) ((except that I don't like it's API, of course)). Thanks everyone! p.s.: it seems people read code much more often then they do read FAQ

3 2

Python memory allocation in embedded interpreters
by Tim Lesher Aug. 25, 2011

Aug. 25, 2011

Issue 3329 (http://bugs.python.org/issue3329, "API for setting the memory allocator used by Python") called for the ability to redirect Python's memory allocation at the lowest level from the C runtime malloc/realloc/free to a user-supplied allocator. The consensus seemed to turn to a set of macros, similar to the (now-deprecated) PyMem_XXXX family of macros, that can redirect to a user-supplied allocator based on a compile-time switch, without forcing an indirection on "normal" builds of … [View More]

1 0

Re: [Python-ideas] Enums
by Spectral One Aug. 20, 2011

Aug. 20, 2011

> Subject: > Re: [Python-ideas] Enums > From: > Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Date: > 8/19/2011 7:31 PM > > To: > python-ideas(a)python.org > > > Spectral One wrote: > >> def Box: >> def SetMake( new_make:enum(pcmake)=ibm ): > > >> b.SetMake(apple) #Works fine > > How do you propose to implement this? > > Currently, function arguments are evaluated without any > knowledge of the function to … [View More]which they are going to be > passed. To what extent would you change that, and how? > > -- > Greg Hmm, is setting a flag to switch twixt normal and another namespace really all that tricky? It would probably help to know how Python handles namespaces already, but I don't. In fact, I don't know Python well enough to do much more than speculate on how it's implemented. I was just figuring that the syntax and API are the harder parts to do well, anyhow, so I suggested in that mode. That NO evaluation is done before entry is a bit of a surprise. Obviously, the number of arguments is stored and checked, since TypeError is raised on argument number mismatches. There are obviously various ways to accomplish this, but the idea was, I confess, unconstrained by implementation details I don't know. Some possibilities that pop to mind: Such things could be held in a list, much like object properties or classes, which is available to be interrogated by the parser. When checking the argument list, it could also check lists for associated enums. If functions are complete objects, you could embed the information in the object for each specified argument in some standard way. 'function.__enums__()' maybe It seems that the function, on call, would need to know to check these things or not, but that's something that could be handled behind the scenes with a block to test that's simply conditionally skipped or added as needed. That would still require that the values be processed differently while being interpreted, unless some global rule was instituted to get around it. (Late-parsing arguments might work, or making sure that the names of variables are passed as strings to the function, so the function can ignore incorrectly-parsed identifiers. Of course THAT would require a tone more processing, or would preclude multiple values, so I really don't like it.) Actually, if our hearts aren't set on passing the enum names as identifiers, we could pass them as strings that are parsed on the other side. (rules for combining them would be part of the standard enum object definition.) This wouldn't look as neat to me, but it would move the namespace issue completely inside the recipient function. A function could fake it or actually use the the enum functions based on string parsing. def Box: def SetMake( new_make=enum(pcmake)["ibm"] ) ... b.SetMake("apple") Beyond that, it's mostly just a matter of determining proper behavior for an enum object type and the events it should raise, etc. (An extended dictionary class which stashes {key:(key,value)} could work as a basis.) -Nate [View Less]

1 0

Re: [Python-ideas] Enums
by Spectral One Aug. 19, 2011

Aug. 19, 2011

A thought occurred after reading through this thread. Given that one of the most common uses of enums is to define the range of valid options to pass and to demystify such magic numbers, it seems that the proper place to have enums' variable names defined is the module in which they are declared and used, but in cases where the meaning is clear, and only there, pull them out. (that is, normally, one needs to type "EnumInstanceName.Code" e.g. "Colors.red", but when in a place where the … [View More]author specified a specific enum, the code is all that's required. e.g. "red") After all, unless someone wants to allow MULTIPLE enums as an argument, the normal collision rules should prevent most of the problems. This should work for either the single-value or multiple value cases, since it merely imports the namespace, and the parser otherwise works as normal. (Such importation would need to override, and pull from the enum, wherever it's called from.) I suppose this could be a general soft-typing syntax, and the Enum object type could simply be a valid instance, but I don't think I like that notion. I'd prefer the syntax to be clearly a set enumeration. Of course, I'd also like some fast and easy way of declaring whether the enum in question should allow aggregate/composite types, or singular values from the list, with an exception raised on failure to comply. something like the following: #exa.py: enum(single) pc_make: #Declare pc_make as an enum type with the following contents. apple ibm=2.5 atari="Just to demonstrate non-numeric" #I picture defaulting to unique, non-overlapping integer values, unless otherwise defined. #Valid types would best be hashable #I suppose "pc_make=enum(single)" is more consistent, if we think of enum as a class, #Also, enum(1) might be better, for defining values accepted. I digress. def Box: def SetMake( new_make:enum(pcmake)=ibm ): #to set the id to "new_make", the enum to pcmake, and the default setting to "ibm" pass #2nd_module.py: from exa import Box b=Box() b.SetMake(apple) #Works fine b.SetMake(ibm + apple) $Raises an enum_error, since the enum was declared singular. (In hindsight, the argument definition is a better place to to define the singular/multiple values option, though having the ability to defining once for all would make use easier.) Hopefully the idea is clear. The repr(enum_code)='enum_code' notion would certainly be compatible. -Nate > That's assuming the data export format doesn't support enums. > > I would fine enums whose str() is just an int pretty annoying. Note > that str(True) == repr(True) == 'True' and that's how I'd like it to > be for enums in general. If you want the int value, use int(e). If you > want the class name, use e.__class__.__name__ (maybe e.__class__ is > what you're after anyway). My observation is that most enums have > sufficiently unique names that during a debugging session the class is > not a big mystery; it's the mapping from value to name that's hard to > memorize. As a compromise, I could live with str(e) == 'red' and > repr(e) == 'Color.red'. But I don't think I could stomach str(e) == 1. > > -- --Guido van Rossum (python.org/~guido) [View Less]

2 1

Re: [Python-ideas] Syntax for dedented strings
by David Townshend Aug. 19, 2011

Aug. 19, 2011

+1 for strings in unit tests, but not so useful for docstrings. But the proposal was as a new string type, not necessarily docstrings. But maybe a string method would be better: >>> """ ... This ... is ... indented.""".deindent() """ This is indented""" On Aug 19, 2011 2:55 PM, "Dag Odenhall" <dag.odenhall(a)gmail.com> wrote: > > On Tue, 2011-08-16 at 23:43 +0100, Michael Foord wrote: > > The place I'm concerned about is the interactive interpreter, … [View More]

1 0

Syntax for dedented strings
by Michael Foord Aug. 19, 2011

Aug. 19, 2011

Hello all, I'm sure this will be shot down [1], but it annoys me often enough that I'm going to suggest it anyway. :-) A recent tweet of Raymond's reminded me how useful textwrap.dedent() is for dedenting triple quoted strings in code blocks: def function(): this_string = textwrap.dedent("""\ Here is some indented text. that dedent will handle for us.""" ) Unfortunately that doesn't work for docstrings as they must be string literals. It is compounded by the fact … [View More]

7 11