Mailman 3 January 2008 - Python-ideas

Ideas towards GIL removal
by Greg Ewing 18 Jul '08

18 Jul '08

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". Any refcount operation on a permanent object would be a no-op, so no locking would be needed. This would also have the benefit of eliminating any need to write to the object's memory at all when it's only being read. 2) Objects owned by a thread Python code creates and destroys temporary objects at a high rate -- stack frames, argument tuples, intermediate results, etc. If the code is executed by a thread, those objects are rarely if ever seen outside of that thread. It would be beneficial if refcount operations on such objects could be carried out by the thread that created them without locking. To achieve this, two extra fields could be added to the object header: an "owning thread id" and a "local reference count". (The existing refcount field will be called the "global reference count" in what follows.) An object created by a thread has its owning thread id set to that thread. When adjusting an object's refcount, if the current thread is the object's owning thread, the local refcount is updated without locking. If the object has no owning thread, or belongs to a different thread, the object is locked and the global refcount is updated. The object is considered garbage only when both refcounts drop to zero. Thus, after a decref, both refcounts would need to be checked to see if they are zero. When decrementing the local refcount and it reaches zero, the global refcount can be checked without locking, since a zero will never be written to it until it truly has zero non-local references remaining. I suspect that these two strategies together would eliminate a very large proportion of refcount-related activities requiring locking, perhaps to the point where those remaining are infrequent enough to make GIL removal practical. -- Greg

8 19

adopt an enum type for the standard library?
by Mark Summerfield 12 Apr '08

12 Apr '08

AFAIK if you want enumerations you must either create your own, or use a third party module, or use namedtuple: Files = collections.namedtuple("Files", "minimum maximum")(1, 200) ... x = Files.minimum Using: MINIMUM, MAXIMUM = 1, 200 is often inconvenient, since you might have several different ones. Of course you could do: MIN_FILES = 1 MIN_DIRS = 0 Personally, I like enums and consider them to be a fundamental, but I don't like the above approaches. There is an enum module in PyPI http://pypi.python.org/pypi/enum/ and there are several versions in the Python Cookbook. Wouldn't one of these be worth adopting for the standard library? -- Mark Summerfield, Qtrac Ltd., www.qtrac.eu

10 14

non-Pre-PEP: Syntactic replacement of built-in types with user-defined callables (Take 2... fewer newlines!)
by Taro 31 Jan '08

31 Jan '08

Hi all... again, Built-in types such as float, string, or list are first-class citizens in Python sourcefiles, having syntactic support: myfloat = 1.0 mystring = "my string" mylist = [1,2,4,8] mydict = {1:"a", 2:"b", 3:"c"} myset = {1,2,3} User-defined classes are second-class citizens, requiring data to be manually converted from a type: mydecimal = Decimal("1.00000000000000001") myrope = Rope("my rope") myblist = BList([1,2,4,8]) myordereddict = OrderedDict((1,"a"), (2, "b"), (3, "c")) myfrozenset = frozenset([1,2,3]) If there's only one or two conversions needed in a file, then such conversion is not particularly burdensome, but if one wants to consistently use (say) decimals throughout a file then the ability to use a literal syntax makes for prettier source. Some languages have open types, allowing the addition of methods to built-in types. This is not considered desired behaviour for Python, since modifications made in one module can potentially affect code in other modules. A typedef is syntactic sugar to allow user-defined replacements to be treated as first-class citizens in code. They affect only the module in which they appear and do not modify the original type. To be typedeffable, something must be a builtin type, have a constant/syntactic representation, and be callable. Hence, typedeffable types would be limited to complex, dict, float, int, ?object?, list, slice, set, string, and tuple. No modification is made to __builtins__ or types, so conversion and/or reference to the original type is still possible. The syntax for a typedef is: from MODULE typedef ADAPTOR as TYPE OR typedef BUILTINADAPTOR as TYPE Syntactic constants of a given type are then wrapped with a call: ADAPTOR(SOURCELITERAL) where SOURCELITERAL is the string that appears in the sourcecode eg: from decimal typedef Decimal as float i = 1.000000000000000000000000000001 translates as: from decimal import Decimal as float i = Decimal("1.000000000000000000000000000001") Syntactic collections of a given type are always provided with a list of objects, eg: from decimal typedef Decimal as float from blist typedef BList as list b = [1.1, 4.2] translates as: from decimal import Decimal as float from blist import BList as list b = Blist([Decimal("1.1"), Decimal("4.2")]) and from collections typedef OrderedDict as dict d = {1:"a", 2:"b", 3:"c"} as: from collections import OrderedDict as dict d = OrderedDict([(1,"a"), (2,"b"), (3,"c")]) A typedef appears at the start of a module immediately after any __future__ imports. As no adaptors can be defined in a module before a typedef and typedefs are in no way a forward declaration, "typedef ADAPTOR as TYPE" only works for builtins, since to do otherwise would lead to one of two unpalatable options; either: a/ definition of adaptors would have to be allowed pre-typedef, which would allow them to be buried in code, making them far easier to miss; or b/ adaptors would be defined after the typedef, which means that you'd have to handle: typedef float as float def float(): pass or: typedef MyObject as object class MyObject(object): pass or: typedef myfloat as float x = 1.1 def myfloat(): pass It is true that if a valid typedef is made, the type can be redefined within the module; but the "consenting adults" rule applies -- it's possible to redefine str and float multiple times within a module as well, but that isn't recommended either (and the typedef at the top of the module at least indicates that non-standard behaviour is to be expected) It is a SyntaxError to typedef the same type more than once: from decimal typedef Decimal as float from types typedef FloatType as float #SyntaxError("Type 'float' already redefined.") Spelling: "typedef" is prettier than "pragma", and less likely to be in use than "use" but its behaviour is somewhat different from C's typedef, so perhaps another term may be preferred. Theoretical Performance Differences: Since a typedef is purely syntactic sugar, and all tranformational work would be done at compilation, running code should be no slower (there should be no new opcodes necessary) and by default no faster that performing manual conversion (though they may assist an optimisation). Unless optimisation magic is available, performance-critical code should be careful when typedeffing not to use typedeffed literals in inner loops. I know it's considered polite to provide code, but any implementation would have to be in C, so please accept this extremely fake dummy implementation which in no way resembles the way things really work as a poor substitute: typedefs = { FLOATTYPE: None, INTTYPE: None, LISTTYPE: None, DICTTYPE: None, ... } while True line = lines.next() type, module, adaptor = parsetypedef(line) if type is None: break if typedefs[type] is not None: raise SyntaxError("Typedef redef") typedefs[type] = adaptor if module is not None: emit_bytecode_for_import_from(type, module, adaptor) else: emit_bytecode_for_assignment(type, adaptor) parse([line] + lines) ... def emit_float(...): if typedefs[FLOATTYPE] is not None: emit_constant(typedefs[FLOATTYPE][0], stringliteral) else: ... # standard behaviour def emit_list(...): if typedefs[LISTTYPE] is not None: emit(LOADGLOBAL, typedefs[LISTTYPE]) # standard behaviour if typedefs[LISTTYPE] is not None: emit(CALL_FUNCTION) # standard behaviour All rights are assigned to the Python Software Foundation

6 8

Re: [Python-ideas] non-Pre-PEP: Syntactic replacement of built-in types with user-defined callables (Take 2... fewer newlines!)
by Raymond Hettinger 31 Jan '08

31 Jan '08

> If I understood your proposal correctly, it's just about > dynamically *replacing* the built-in types (with their > respective syntax) with some type of your own. FWIW, there is an interesting type replacement recipe on the bottom of the page at: http://docs.python.org/lib/module-tokenize.html Raymond

1 0

Re: [Python-ideas] non-Pre-PEP: Syntactic replacement of built-in pes with user-defined callables.
by Matt Chisholm 28 Jan '08

28 Jan '08

I don't think I followed all of what you wrote. However, it would certainly be convenient shorthand to be able to say that, for an entire file, program, or module, the float literal syntax created Decimals instead. And I can imagine cases where an entire app or module was always using ordered dicts instead of dicts, or frozensets instead of sets. -matt

1 0

non-Pre-PEP: Syntactic replacement of built-in types with user-defined callables.
by Taro 28 Jan '08

28 Jan '08

Hi all, Built-in types such as float, string, or list are first-class citizens in Python sourcefiles, having syntactic support: myfloat = 1.0 mystring = "my string" mylist = [1,2,4,8] mydict = {1:"a", 2:"b", 3:"c"} myset = {1,2,3} User-defined classes are second-class citizens, requiring data to be manually converted from a type: mydecimal = Decimal("1.00000000000000001") myrope = Rope("my rope") myblist = BList([1,2,4,8]) myordereddict = OrderedDict((1,"a"), (2, "b"), (3, "c")) myfrozenset = frozenset([1,2,3]) If there's only one or two conversions needed in a file, then such conversion is not particularly burdensome, but if one wants to consistently use (say) decimals throughout a file then the ability to use a literal syntax makes for prettier source. Some languages have open types, allowing the addition of methods to built-in types. This is not considered desired behaviour for Python, since modifications made in one module can potentially affect code in other modules. A typedef is syntactic sugar to allow user-defined replacements to be treated as first-class citizens in code. They affect only the module in which they appear and do not modify the original type. To be typedeffable, something must be a builtin type, have a constant/syntactic representation, and be callable. Hence, typedeffable types would be limited to complex, dict, float, int, ?object?, list, slice, set, string, and tuple. No modification is made to __builtins__ or types, so conversion and/or reference to the original type is still possible. The syntax for a typedef is: from MODULE typedef ADAPTOR as TYPE OR typedef BUILTINADAPTOR as TYPE Syntactic constants of a given type are then wrapped with a call: ADAPTOR(SOURCELITERAL) where SOURCELITERAL is the string that appears in the sourcecode eg: from decimal typedef Decimal as float i = 1.000000000000000000000000000001 translates as: from decimal import Decimal as float i = Decimal("1.000000000000000000000000000001") Syntactic collections of a given type are always provided with a list of objects, eg: from decimal typedef Decimal as float from blist typedef BList as list i = 1.000000000000000000000000000001 b = [1.1, 4.2] translates as: from decimal import Decimal as float from blist import BList as list b = Blist([Decimal("1.1"), Decimal("4.2")]) and from collections typedef OrderedDict as dict d = {1:"a", 2:"b", 3:"c"} as: from collections import OrderedDict as dict d = OrderedDict([(1,"a"), (2,"b"), (3,"c")]) A typedef appears at the start of a module immediately after any __future__ imports. As no adaptors can be defined in a module before a typedef and typedefs are in no way a forward declaration, "typedef ADAPTOR as TYPE" only works for builtins, since to do otherwise would lead to one of two unpalatable options; either: a/ definition of adaptors would have to be allowed pre-typedef, which would allow them to be buried in code, making them far easier to miss; or b/ adaptors would be defined after the typedef, which means that you'd have to handle: typedef float as float def float(): pass or: typedef MyObject as object class MyObject(object): pass or: typedef myfloat as float x = 1.1 def myfloat(): pass It is true that if a valid typedef is made, the type can be redefined within the module; but the "consenting adults" rule applies -- it's possible to redefine str and float multiple times within a module as well, but that isn't recommended either (and the typedef at the top of the module at least indicates that non-standard behaviour is to be expected) It is a SyntaxError to typedef the same type more than once: from decimal typedef Decimal as float from types typedef FloatType as float #SyntaxError("Type 'float' already redefined.") Spelling: "typedef" is prettier than "pragma", and less likely to be in use than "use" but its behaviour is somewhat different from C's typedef, so perhaps another term may be preferred. Theoretical Performance Differences: Since a typedef is purely syntactic sugar, and all tranformational work would be done at compilation, running code should be no slower (there should be no new opcodes necessary) and by default no faster that performing manual conversion (though they may assist an optimisation). Unless optimisation magic is available, performance-critical code should be careful when typedeffing not to use typedeffed literals in inner loops. I know it's considered polite to provide code, but any implementation would have to be in C, so please accept this extremely fake dummy implementation which in no way resembles the way things really work as a poor substitute: typedefs = { FLOATTYPE: None, INTTYPE: None, LISTTYPE: None, DICTTYPE: None, ... } while True line = lines.next() type, module, adaptor = parsetypedef(line) if type is None: break if typedefs[type] is not None: raise SyntaxError("Typedef redef") typedefs[type] = adaptor if module is not None: emit_bytecode_for_import_from(type, module, adaptor) else: emit_bytecode_for_assignment(type, adaptor) parse([line] + lines) ... def emit_float(...): if typedefs[FLOATTYPE] is not None: emit_constant(typedefs[FLOATTYPE][0], stringliteral) else: ... # standard behaviour def emit_list(...): if typedefs[LISTTYPE] is not None: emit(LOADGLOBAL, typedefs[LISTTYPE]) # standard behaviour if typedefs[LISTTYPE] is not None: emit(CALL_FUNCTION) # standard behaviour All rights are assigned to the Python Software Foundation

1 0

More on set literals
by VanL 26 Jan '08

26 Jan '08

I like Raymond's suggestion (apparently adopted) that {1,2,3} is a frozenset literal. However, I also like the {{1,2,3}} syntax. So I would propose that {{}} become the literal for the *mutable* set, not the frozenset. So: frozenset() # empty frozenset {1,2,3} # frozenset literal {} # empty dict {1:2, 3:4} #dict literal {{}} (or set()) # empty mutable set {{1,2,3}} # set literal My rationale is as follows: 1. It visually distinguishes sets and frozensets, without making them take up a lot of space. If people see the {{}}, they will be reminded that this set is mutable. 2. In Raymond's example, his point was that most people don't want a mutable literal - they would be better served by a frozenset. However, sometimes you *do* want to add elements to a set literal. For example: # I am parsing the configuration file for a pure-python webserver. # The config file allows me to add new filetype extensions # that will be served. # Default HTML extensions: HTML_EXTS = {{'.html', '.htm'}} # later, when going through config file: if filetype.handler == HTMLHandler: HTML_EXTS.add(filetype.ext) I know that this can be done other ways (set(['.html', '.htm'])), but I like the way this looks. Thanks, Van P.S.: The bikeshed should be yellow.

5 4

A number is a number
by Neil Toronto 26 Jan '08

26 Jan '08

Isn't it? Python 3.0 integers are just integers. Now ceil and floor return ints from float arguments. It seems like things are moving closer to just having a single "number" type. For most applications, because of duck typing, you'd never know or care whether you're punting a float, int, or complex number around, and you'd usually not care. Why not just go whole-hog and unify them all? Neil

2 1

Re: [Python-ideas] An easier syntax for writing decorators(&similar things)?
by Steven Bethard 25 Jan '08

25 Jan '08

On Jan 24, 2008 3:26 PM, Aaron Brady <castironpi(a)comcast.net> wrote: > > I don't consider that a use case, or real code. ;-) Yes, you can > > construct curry with it. But what do you want to use curry for? Show > > me some actual Python packages that use the curry function (or your > > prepartial function) and then we can talk. > > Original function: > 1. urlparse( urlstring[, default_scheme[, allow_fragments]]) > 2. urlretrieve( url[, filename[, reporthook[, data]]]) > > Prepartial in action: > 1. parseA= prepartial( 'ftp', False ) > 2. retrieveA= prepartial( 'temp.htm', callbackA ) > > Equivalent: > 1. parseAB= partial( default_scheme= 'ftp', allow_fragments= True ) > 2. retrieveAB= partial( filename= 'temp.htm', reporthook= callbackA ) This is closer to what I'm asking for but these are still not instances of real code[1]. To build a convincing argument for a new language feature, you need to show not only a theoretical use case, but a practical one. In general, that means you need to show that it is a frequent need in a large code base. Finding a bunch of examples of it in the standard library or big applications like Zope or Twisted would be a good start. This is something that's expected of any PEP-worthy idea. [1] I know you didn't copy these from real code anywhere because they all have typos -- they're missing urlparse/urlretrieve as the first arguments to partial() or prepartial(). STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

2 1

Re: [Python-ideas] An easier syntax for writing decorators (& similar things)?
by Aaron Brady 24 Jan '08

24 Jan '08

> -----Original Message----- > On 8 Oct 2007, at 10:57, Arnaud Delobelle wrote: > > > > On Mon, October 8, 2007 4:33 am, Adam Atlas wrote: > >> When writing decorators especially when it's one that needs arguments > >> other than the function to be wrapped, it often gets rather ugly... > [...] > > Whhy not create a (meta-)decorator to do this? Something like: > [...] Following up post from 10/8/07. > To follow up on my untested suggestion, here's one that is tested: > > # This metadecorator hasn't changed > > def decorator_withargs(decf): > def decorator(*args, **kwargs): > def decorated(f): > return decf(f, *args, **kwargs) > return decorated > return decorator This is equivalent to: (1) decorator_withargs= partial( partial, prepartial ) , where prepartial is roughly the same as partial as you might expect: (2) 1 def prepartial(func, *args, **keywords): 2 def newfunc(*fargs, **fkeywords): 3 newkeywords = keywords.copy() 4 newkeywords.update(fkeywords) 5 return func(*(fargs+ args), **newkeywords) 6 newfunc.func = func 7 newfunc.args = args 8 newfunc.keywords = keywords 9 return newfunc Partial is the same outside of line 5: (3) 5 return func(*(args + fargs), **newkeywords) Results are the same: -> f 1 f -> 2 2 Intriguing. > # Here's how to use it to create a decorator > > @decorator_withargs > def mydec(f, before='entering %s', after='%s returns %%s'): > before = before % f.__name__ > after = after % f.__name__ > def decorated(*args, **kwargs): > print before > result = f(*args, **kwargs) > print after % result > return result > return decorated > > > # Now I can decorate a function with my new decorator > > @mydec(before='-> %s', after='%s -> %%s') > def f(x): > print x > return x+1 > > > Then > > >>> f(1) > -> f > 1 > f -> 2 > 2 > > -- > Arnaud

4 16