I've been thinking about some ideas for reducing the
amount of refcount adjustment that needs to be done,
with a view to making GIL removal easier.
1) Permanent objects
In a typical Python program there are many objects
that are created at the beginning and exist for the
life of the program -- classes, functions, literals,
etc. Refcounting these is a waste of effort, since
they're never going to go away.
So perhaps there could be a way of marking such
objects as "permanent" or "immortal". Any refcount
operation on a permanent object would be a no-op,
so no locking would be needed. This would also have
the benefit of eliminating any need to write to the
object's memory at all when it's only being read.
2) Objects owned by a thread
Python code creates and destroys temporary objects
at a high rate -- stack frames, argument tuples,
intermediate results, etc. If the code is executed
by a thread, those objects are rarely if ever seen
outside of that thread. It would be beneficial if
refcount operations on such objects could be carried
out by the thread that created them without locking.
To achieve this, two extra fields could be added
to the object header: an "owning thread id" and a
"local reference count". (The existing refcount
field will be called the "global reference count"
in what follows.)
An object created by a thread has its owning thread
id set to that thread. When adjusting an object's
refcount, if the current thread is the object's owning
thread, the local refcount is updated without locking.
If the object has no owning thread, or belongs to
a different thread, the object is locked and the
global refcount is updated.
The object is considered garbage only when both
refcounts drop to zero. Thus, after a decref, both
refcounts would need to be checked to see if they
are zero. When decrementing the local refcount and
it reaches zero, the global refcount can be checked
without locking, since a zero will never be written
to it until it truly has zero non-local references
remaining.
I suspect that these two strategies together would
eliminate a very large proportion of refcount-related
activities requiring locking, perhaps to the point
where those remaining are infrequent enough to make
GIL removal practical.
--
Greg
AFAIK if you want enumerations you must either create your own, or use a
third party module, or use namedtuple:
Files = collections.namedtuple("Files", "minimum maximum")(1, 200)
...
x = Files.minimum
Using:
MINIMUM, MAXIMUM = 1, 200
is often inconvenient, since you might have several different ones. Of
course you could do:
MIN_FILES = 1
MIN_DIRS = 0
Personally, I like enums and consider them to be a fundamental, but I
don't like the above approaches.
There is an enum module in PyPI
http://pypi.python.org/pypi/enum/
and there are several versions in the Python Cookbook.
Wouldn't one of these be worth adopting for the standard library?
--
Mark Summerfield, Qtrac Ltd., www.qtrac.eu
Hi all... again,
Built-in types such as float, string, or list are first-class citizens
in Python sourcefiles, having syntactic support:
myfloat = 1.0
mystring = "my string"
mylist = [1,2,4,8]
mydict = {1:"a", 2:"b", 3:"c"}
myset = {1,2,3}
User-defined classes are second-class citizens, requiring data to be
manually converted from a type:
mydecimal = Decimal("1.00000000000000001")
myrope = Rope("my rope")
myblist = BList([1,2,4,8])
myordereddict = OrderedDict((1,"a"), (2, "b"), (3, "c"))
myfrozenset = frozenset([1,2,3])
If there's only one or two conversions needed in a file, then such
conversion is not particularly burdensome, but if one wants to
consistently use (say) decimals throughout a file then the ability to
use a literal syntax makes for prettier source.
Some languages have open types, allowing the addition of methods to
built-in types. This is not considered desired behaviour for Python,
since modifications made in one module can potentially affect code in
other modules.
A typedef is syntactic sugar to allow user-defined replacements to be
treated as first-class citizens in code. They affect only the module
in which they appear and do not modify the original type. To be
typedeffable, something must be a builtin type, have a
constant/syntactic representation, and be callable. Hence,
typedeffable types would be limited to complex, dict, float, int,
?object?, list, slice, set, string, and tuple. No modification is
made to __builtins__ or types, so conversion and/or reference to the
original type is still possible.
The syntax for a typedef is:
from MODULE typedef ADAPTOR as TYPE
OR
typedef BUILTINADAPTOR as TYPE
Syntactic constants of a given type are then wrapped with a call:
ADAPTOR(SOURCELITERAL)
where SOURCELITERAL is the string that appears in the sourcecode
eg:
from decimal typedef Decimal as float
i = 1.000000000000000000000000000001
translates as:
from decimal import Decimal as float
i = Decimal("1.000000000000000000000000000001")
Syntactic collections of a given type are always provided with a list
of objects, eg:
from decimal typedef Decimal as float
from blist typedef BList as list
b = [1.1, 4.2]
translates as:
from decimal import Decimal as float
from blist import BList as list
b = Blist([Decimal("1.1"), Decimal("4.2")])
and
from collections typedef OrderedDict as dict
d = {1:"a", 2:"b", 3:"c"}
as:
from collections import OrderedDict as dict
d = OrderedDict([(1,"a"), (2,"b"), (3,"c")])
A typedef appears at the start of a module immediately after any
__future__ imports. As no adaptors can be defined in a module before
a typedef and typedefs are in no way a forward declaration, "typedef
ADAPTOR as TYPE" only works for builtins, since to do otherwise would
lead to one of two unpalatable options; either:
a/ definition of adaptors would have to be allowed pre-typedef, which
would allow them to be buried in code, making them far easier to miss;
or
b/ adaptors would be defined after the typedef, which means that you'd
have to handle:
typedef float as float
def float():
pass
or:
typedef MyObject as object
class MyObject(object):
pass
or:
typedef myfloat as float
x = 1.1
def myfloat():
pass
It is true that if a valid typedef is made, the type can be redefined
within the module; but the "consenting adults" rule applies -- it's
possible to redefine str and float multiple times within a module as
well, but that isn't recommended either (and the typedef at the top of
the module at least indicates that non-standard behaviour is to be
expected)
It is a SyntaxError to typedef the same type more than once:
from decimal typedef Decimal as float
from types typedef FloatType as float #SyntaxError("Type 'float'
already redefined.")
Spelling: "typedef" is prettier than "pragma", and less likely to be
in use than "use" but its behaviour is somewhat different from C's
typedef, so perhaps another term may be preferred.
Theoretical Performance Differences: Since a typedef is purely
syntactic sugar, and all tranformational work would be done at
compilation, running code should be no slower (there should be no new
opcodes necessary) and by default no faster that performing manual
conversion (though they may assist an optimisation). Unless
optimisation magic is available, performance-critical code should be
careful when typedeffing not to use typedeffed literals in inner
loops.
I know it's considered polite to provide code, but any implementation
would have to be in C, so please accept this extremely fake dummy
implementation which in no way resembles the way things really work as
a poor substitute:
typedefs = { FLOATTYPE: None,
INTTYPE: None,
LISTTYPE: None,
DICTTYPE: None,
... }
while True
line = lines.next()
type, module, adaptor = parsetypedef(line)
if type is None:
break
if typedefs[type] is not None:
raise SyntaxError("Typedef redef")
typedefs[type] = adaptor
if module is not None:
emit_bytecode_for_import_from(type, module, adaptor)
else:
emit_bytecode_for_assignment(type, adaptor)
parse([line] + lines)
...
def emit_float(...):
if typedefs[FLOATTYPE] is not None:
emit_constant(typedefs[FLOATTYPE][0], stringliteral)
else:
... # standard behaviour
def emit_list(...):
if typedefs[LISTTYPE] is not None:
emit(LOADGLOBAL, typedefs[LISTTYPE])
# standard behaviour
if typedefs[LISTTYPE] is not None:
emit(CALL_FUNCTION)
# standard behaviour
All rights are assigned to the Python Software Foundation
> If I understood your proposal correctly, it's just about
> dynamically *replacing* the built-in types (with their
> respective syntax) with some type of your own.
FWIW, there is an interesting type replacement recipe on
the bottom of the page at:
http://docs.python.org/lib/module-tokenize.html
Raymond
I don't think I followed all of what you wrote. However, it would
certainly be convenient shorthand to be able to say that, for an
entire file, program, or module, the float literal syntax created
Decimals instead.
And I can imagine cases where an entire app or module was always using
ordered dicts instead of dicts, or frozensets instead of sets.
-matt
Hi all,
Built-in types such as float, string, or list are first-class citizens in
Python sourcefiles, having syntactic support:
myfloat = 1.0
mystring = "my string"
mylist = [1,2,4,8]
mydict = {1:"a", 2:"b", 3:"c"}
myset = {1,2,3}
User-defined classes are second-class citizens, requiring data to be
manually converted from a type:
mydecimal = Decimal("1.00000000000000001")
myrope = Rope("my rope")
myblist = BList([1,2,4,8])
myordereddict = OrderedDict((1,"a"), (2, "b"), (3, "c"))
myfrozenset = frozenset([1,2,3])
If there's only one or two conversions needed in a file, then such
conversion is not particularly burdensome, but if one wants to consistently
use (say) decimals throughout a file then the ability to use a literal
syntax makes for prettier source.
Some languages have open types, allowing the addition of methods to built-in
types. This is not considered desired behaviour for Python, since
modifications made in one module can potentially affect code in other
modules.
A typedef is syntactic sugar to allow user-defined replacements to be
treated as first-class citizens in code. They affect only the module in
which they appear and do not modify the original type. To be typedeffable,
something must be a builtin type, have a constant/syntactic representation,
and be callable. Hence, typedeffable types would be limited to complex,
dict, float, int, ?object?, list, slice, set, string, and tuple. No
modification is made to __builtins__ or types, so conversion and/or
reference to the original type is still possible.
The syntax for a typedef is:
from MODULE typedef ADAPTOR as TYPE
OR
typedef BUILTINADAPTOR as TYPE
Syntactic constants of a given type are then wrapped with a call:
ADAPTOR(SOURCELITERAL)
where SOURCELITERAL is the string that appears in the sourcecode
eg:
from decimal typedef Decimal as float
i = 1.000000000000000000000000000001
translates as:
from decimal import Decimal as float
i = Decimal("1.000000000000000000000000000001")
Syntactic collections of a given type are always provided with a list of
objects, eg:
from decimal typedef Decimal as float
from blist typedef BList as list
i = 1.000000000000000000000000000001
b = [1.1, 4.2]
translates as:
from decimal import Decimal as float
from blist import BList as list
b = Blist([Decimal("1.1"), Decimal("4.2")])
and
from collections typedef OrderedDict as dict
d = {1:"a", 2:"b", 3:"c"}
as:
from collections import OrderedDict as dict
d = OrderedDict([(1,"a"), (2,"b"), (3,"c")])
A typedef appears at the start of a module immediately after any __future__
imports. As no adaptors can be defined in a module before a typedef and
typedefs are in no way a forward declaration, "typedef ADAPTOR as TYPE" only
works for builtins, since to do otherwise would lead to one of two
unpalatable options; either:
a/ definition of adaptors would have to be allowed pre-typedef, which would
allow them to be buried in code, making them far easier to miss; or
b/ adaptors would be defined after the typedef, which means that you'd have
to handle:
typedef float as float
def float():
pass
or:
typedef MyObject as object
class MyObject(object):
pass
or:
typedef myfloat as float
x = 1.1
def myfloat():
pass
It is true that if a valid typedef is made, the type can be redefined within
the module; but the "consenting adults" rule applies -- it's possible to
redefine str and float multiple times within a module as well, but that
isn't recommended either (and the typedef at the top of the module at least
indicates that non-standard behaviour is to be expected)
It is a SyntaxError to typedef the same type more than once:
from decimal typedef Decimal as float
from types typedef FloatType as float #SyntaxError("Type 'float'
already redefined.")
Spelling: "typedef" is prettier than "pragma", and less likely to be in use
than "use" but its behaviour is somewhat different from C's typedef, so
perhaps another term may be preferred.
Theoretical Performance Differences: Since a typedef is purely syntactic
sugar, and all tranformational work would be done at compilation, running
code should be no slower (there should be no new opcodes necessary) and by
default no faster that performing manual conversion (though they may assist
an optimisation). Unless optimisation magic is available,
performance-critical code should be careful when typedeffing not to use
typedeffed literals in inner loops.
I know it's considered polite to provide code, but any implementation would
have to be in C, so please accept this extremely fake dummy implementation
which in no way resembles the way things really work as a poor substitute:
typedefs = { FLOATTYPE: None,
INTTYPE: None,
LISTTYPE: None,
DICTTYPE: None,
... }
while True
line = lines.next()
type, module, adaptor = parsetypedef(line)
if type is None:
break
if typedefs[type] is not None:
raise SyntaxError("Typedef redef")
typedefs[type] = adaptor
if module is not None:
emit_bytecode_for_import_from(type, module, adaptor)
else:
emit_bytecode_for_assignment(type, adaptor)
parse([line] + lines)
...
def emit_float(...):
if typedefs[FLOATTYPE] is not None:
emit_constant(typedefs[FLOATTYPE][0], stringliteral)
else:
... # standard behaviour
def emit_list(...):
if typedefs[LISTTYPE] is not None:
emit(LOADGLOBAL, typedefs[LISTTYPE])
# standard behaviour
if typedefs[LISTTYPE] is not None:
emit(CALL_FUNCTION)
# standard behaviour
All rights are assigned to the Python Software Foundation
I like Raymond's suggestion (apparently adopted) that {1,2,3} is a
frozenset literal. However, I also like the {{1,2,3}} syntax. So I would
propose that {{}} become the literal for the *mutable* set, not the
frozenset.
So:
frozenset() # empty frozenset
{1,2,3} # frozenset literal
{} # empty dict
{1:2, 3:4} #dict literal
{{}} (or set()) # empty mutable set
{{1,2,3}} # set literal
My rationale is as follows:
1. It visually distinguishes sets and frozensets, without making them
take up a lot of space. If people see the {{}}, they will be reminded
that this set is mutable.
2. In Raymond's example, his point was that most people don't want a
mutable literal - they would be better served by a frozenset. However,
sometimes you *do* want to add elements to a set literal. For example:
# I am parsing the configuration file for a pure-python webserver.
# The config file allows me to add new filetype extensions
# that will be served.
# Default HTML extensions:
HTML_EXTS = {{'.html', '.htm'}}
# later, when going through config file:
if filetype.handler == HTMLHandler:
HTML_EXTS.add(filetype.ext)
I know that this can be done other ways (set(['.html', '.htm'])), but I
like the way this looks.
Thanks,
Van
P.S.: The bikeshed should be yellow.
Isn't it?
Python 3.0 integers are just integers. Now ceil and floor return ints
from float arguments. It seems like things are moving closer to just
having a single "number" type. For most applications, because of duck
typing, you'd never know or care whether you're punting a float, int, or
complex number around, and you'd usually not care.
Why not just go whole-hog and unify them all?
Neil
On Jan 24, 2008 3:26 PM, Aaron Brady <castironpi(a)comcast.net> wrote:
> > I don't consider that a use case, or real code. ;-) Yes, you can
> > construct curry with it. But what do you want to use curry for? Show
> > me some actual Python packages that use the curry function (or your
> > prepartial function) and then we can talk.
>
> Original function:
> 1. urlparse( urlstring[, default_scheme[, allow_fragments]])
> 2. urlretrieve( url[, filename[, reporthook[, data]]])
>
> Prepartial in action:
> 1. parseA= prepartial( 'ftp', False )
> 2. retrieveA= prepartial( 'temp.htm', callbackA )
>
> Equivalent:
> 1. parseAB= partial( default_scheme= 'ftp', allow_fragments= True )
> 2. retrieveAB= partial( filename= 'temp.htm', reporthook= callbackA )
This is closer to what I'm asking for but these are still not
instances of real code[1]. To build a convincing argument for a new
language feature, you need to show not only a theoretical use case,
but a practical one. In general, that means you need to show that it
is a frequent need in a large code base. Finding a bunch of examples
of it in the standard library or big applications like Zope or Twisted
would be a good start. This is something that's expected of any
PEP-worthy idea.
[1] I know you didn't copy these from real code anywhere because they
all have typos -- they're missing urlparse/urlretrieve as the first
arguments to partial() or prepartial().
STeVe
--
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
--- Bucky Katt, Get Fuzzy
> -----Original Message-----
> On 8 Oct 2007, at 10:57, Arnaud Delobelle wrote:
> >
> > On Mon, October 8, 2007 4:33 am, Adam Atlas wrote:
> >> When writing decorators especially when it's one that needs arguments
> >> other than the function to be wrapped, it often gets rather ugly...
> [...]
> > Whhy not create a (meta-)decorator to do this? Something like:
> [...]
Following up post from 10/8/07.
> To follow up on my untested suggestion, here's one that is tested:
>
> # This metadecorator hasn't changed
>
> def decorator_withargs(decf):
> def decorator(*args, **kwargs):
> def decorated(f):
> return decf(f, *args, **kwargs)
> return decorated
> return decorator
This is equivalent to:
(1)
decorator_withargs= partial( partial, prepartial )
, where prepartial is roughly the same as partial as you might expect:
(2)
1 def prepartial(func, *args, **keywords):
2 def newfunc(*fargs, **fkeywords):
3 newkeywords = keywords.copy()
4 newkeywords.update(fkeywords)
5 return func(*(fargs+ args), **newkeywords)
6 newfunc.func = func
7 newfunc.args = args
8 newfunc.keywords = keywords
9 return newfunc
Partial is the same outside of line 5:
(3)
5 return func(*(args + fargs), **newkeywords)
Results are the same:
-> f
1
f -> 2
2
Intriguing.
> # Here's how to use it to create a decorator
>
> @decorator_withargs
> def mydec(f, before='entering %s', after='%s returns %%s'):
> before = before % f.__name__
> after = after % f.__name__
> def decorated(*args, **kwargs):
> print before
> result = f(*args, **kwargs)
> print after % result
> return result
> return decorated
>
>
> # Now I can decorate a function with my new decorator
>
> @mydec(before='-> %s', after='%s -> %%s')
> def f(x):
> print x
> return x+1
>
>
> Then
>
> >>> f(1)
> -> f
> 1
> f -> 2
> 2
>
> --
> Arnaud
