Python IS slow ! [was] Re: Python too slow for real world

[Michael Hudson]

Christian Tismer <tismer at appliedbiometrics.com> writes:

> Randall Hopper wrote:
> > 
> > Christian Tismer:
> >  |Terry Reedy wrote:
> >  |> A batch-mode optimizer analyzing an entire file (module) should be able to
> >  |> detect whether or not function names are rebound.
> >  |>
> >  |> Perhaps module bindings should be considered immutable from outside the
> >  |> module unless explicitly declared otherwise.
> >  |
> >  |I'm thinking of no new keyword, but a mechanism which allows me to lock a
> >  |namespace somehow.
> > 
> > I like this idea in concept.  Though I would prefer a way to have
> > namespaces "lock by default".  Examples:  After a class definition, the
> > class function dictionary is locked.  After a module is fully read, all
> > references are bound and the module namespace is locked.  etc.
> 
> Well, I wouldn't do that by default. By default, everything 
> could stay as it is. First of all, this would not break any
> existing code. Then, many people will want to
> fine tune their modules, and they are perhaps not done
> after a class definition was ready.
> 
> Then, what would you do with classes which depend on each 
> other? You cannot lock them immediately, this would fail.
> Locking them after they both are defined is fine, since
> everything is defined then. With minimum effort and no
> language changes, this will be needed.
> 
> Then think of all the more difficult systems which need
> more effort to become configured. The xml parsers together
> with SAX are an example. If I wanted to lock this, then
> this must be done with care. One would also not lock the mixin
> classes, but only the final workhorse class, bound with
> the correctly selected parser, and so on.
> 
> It might also be necessary to find a way to specify which
> attributes may be locked and which not, since there exist
> indeed cases where Python's super-flexibility is needed :-)
> 
> Depending on how exactly will be implemented, a single line
> at the end of a module should suffice to accomplish this stuff
> for the standard cases. Fine adjustment would take a little more.
> As a side effect, locking a module would also find all 
> referenced but undefined symbols.
> 
> Anyway, this is still no cakewalk and quite a lot of code
> is involved. Needs much more thinking...

I think I can do this. Not to classes yet, but given a function, I can
make you another function where all the global lookups are bound to
the values found at that moment.

It's used like this:

>>> def f(x):
...  return x+y
...
>>> import closure
>>> g=closure.bind(f,y=1)
>>> f(1)
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in f
NameError: y
>>> g(1)
2

It's then easy to write a function that binds all the variables found
in the current environment:

def bind_now(func):
    try:
        raise ""
    except:
        import sys
        frame=sys.exc_traceback.tb_frame.f_back
        l=apply(bind,(func,),frame.f_locals)
        g=apply(bind,(l,),frame.f_globals)    
        return g

This gets used like so:

>>> import closure
>>> y=1
>>> def f(x):
...  return x+y
...
>>> f(0)
1
>>> g=closure.bind_now (f)
>>> y=2
>>> f(0)
2
>>> g(0)
1
>>> 

Is this what you wanted?

A word of warning: this code is nasty, *nasty*, NASTY.  Possibly the
most horrible thing you will see perpetrated in Python this year. It
applies regular expressions to strings of bytecode...

I made Python core repeatedly when debugging it.

However, it works. The returned functions are very fast. I wrote this
package because I wanted to avoid both the tackiness of the `default
argument hack' and the performance penalty of using classes to fake
closures.

As to `sealing' classes in this fashion, I guess it could be
done. You'd need to look for patterns of LOAD_FAST 0 (the first
argument is the zeroth local) followed by LOAD_ATTR. You could then
calculate this and insert a LOAD_CONST instead. The thing is, this
would replace two argumented bytecode with one, changing the length of
the codestring and you'd need to recompute jumps. I haven't had the
bloody-mindedness to get this to work yet.

Code follows...

HTH
Michael

import new,string,re

def copy_code_with_changes(codeobject,
                           argcount=None,
                           nlocals=None,
                           stacksize=None,
                           flags=None,
                           code=None,
                           consts=None,
                           names=None,
                           varnames=None,
                           filename=None,
                           name=None,
                           firstlineno=None,
                           lnotab=None):
    if argcount    is None: argcount    = codeobject.co_argcount
    if nlocals     is None: nlocals     = codeobject.co_nlocals
    if stacksize   is None: stacksize   = codeobject.co_stacksize
    if flags       is None: flags       = codeobject.co_flags
    if code        is None: code        = codeobject.co_code
    if consts      is None: consts      = codeobject.co_consts
    if names       is None: names       = codeobject.co_names
    if varnames    is None: varnames    = codeobject.co_varnames
    if filename    is None: filename    = codeobject.co_filename
    if name        is None: name        = codeobject.co_name
    if firstlineno is None: firstlineno = codeobject.co_firstlineno
    if lnotab      is None: lnotab      = codeobject.co_lnotab
    return new.code(argcount,
                    nlocals,
                    stacksize,
                    flags,
                    code,
                    consts,
                    names,
                    varnames,
                    filename,
                    name,
                    firstlineno,
                    lnotab)

def encode_16(n):
    return '\\%03o\\%03o'%(n%256,n/256)

LOAD_CONST=chr(100)
LOAD_GLOBAL=chr(116)

def munge_code(code,vars):
    codestring=code.co_code
    names=list(code.co_names)
    consts=list(code.co_consts)
    for var,value in vars.items():
        try:
            index=names.index(var)
        except ValueError:
            continue
        codestring=re.sub(LOAD_GLOBAL+encode_16(index),
                          LOAD_CONST+encode_16(len(consts)),
                          codestring)
        consts.append(value)
    return copy_code_with_changes(
        code,
        consts=tuple(consts),
        code=codestring) 

def bind(func,**vars):
    newfunc=new.function(
        munge_code(func.func_code,vars),
        func.func_globals,
        func.func_name)
    newfunc.__doc__=func.__doc__
    newfunc.func_defaults=func.func_defaults
    newfunc.func_doc=func.func_doc
    return newfunc

def bind_locals(func):
    try:
        raise ""
    except:
        import sys
        frame=sys.exc_traceback.tb_frame.f_back
        l=apply(bind,(func,),frame.f_locals)
        frame=None
        return l

def bind_now(func):
    try:
        raise ""
    except:
        import sys
        frame=sys.exc_traceback.tb_frame.f_back
        l=apply(bind,(func,),frame.f_locals)
        g=apply(bind,(l,),frame.f_globals)    
        return g

## examples

def make_adder(n):
    def adder(x):
        return x+n
    return bind_locals(adder)

def make_balance(initial_amount):
    def withdraw(amount):
        if current[0]<amount:
            raise "debt!"
        else:
            current[0]=current[0]-amount
        return current[0]
    return bind