[Python-ideas] Symbolic expressions (or: partials and closures from the inside out)

Terry Reedy tjreedy at udel.edu
Fri Jan 13 00:03:17 CET 2012


On 1/12/2012 3:45 PM, Nathan Rice wrote:
> Greetings,
>
> I have been writing a lot of code lately that involves creating
> symbolic expressions of one form or another, which are then fully
> evaluated at a later time.  Examples of this include Elementwise,

Python is designed for concrete, rather than symbolic computation. But 
the latter has been done.

> where I create expressions that act on every member of an iterable
> (there is a much improved new version coming soon, by the way), and a
> design by contract/validation lib I'm working on (which shall remain
> nameless :D) that uses symbolic expressions in the *args of the
> metaclass __new__ method to generate a constraint class which
> validates input using __instancecheck__.  I do most of this with
> lambdas, a little hacking with closures and FunctionType(), and
> chainable objects.  I am very impressed that python is this flexible,
> but there are some issues with the approach that I would like to
> rectify, namely:
>
> 1.  Because of the early binding behavior of most things in Python,

I think you may be confusing name-resolution time with execution time. 
They are distinct, though somewhat tied together in Python. For example: 
if I hand you a book and say "Read this", I intend that you immediately 
resolve 'this' to the book I just handed you and immediately 'read' it. 
If I instead say, "Tomorrow morning, read this", I still intend that you 
immediately resolve 'this' to a particular object, but now intend that 
you package the object with the action for later execution. I believe 
you want this this mixture of resolution now with action later.

If so,  your problem is that executing 'read(this)' or 'this.read()' 
does both things now, while "def f(): read(this)" or "lambda: 
read(this)" puts off both things until later.

Python has several ways to binds objects now with actions later. Let 
<book> be a reference to 'the book Terry handed me, which I stored 
wherever'.

0: rewrite the text -- a bit awkward in Python.

action = compile("read({this})".format(this=<book>), 'xxx', 'eval')

1. default args -- has to be done when the function is defined.

def action(this = <book>): read(this)

2. closures (nested functions) -- also requires a planned-ahead definition.

make_read(x):
   return lambda: read(x)
action = make_read(<book>)

3. bound methods -- only works for classes with methods.

Class Book:
     def read(self): pass
action = Book(<book>).read

4. partial binding of function params -- generalizes bound methods; 
works for any function and argument.

from functools import partial
action = partial(read, <book>)

> if I want to include isinstance(X, someclass) in a symbolic expression,

Consider using partial, which can totally bind all needed args *now* for 
*later* action.

 >>> from functools import partial
 >>> t = partial(isinstance, 1, int)
 >>> t()
True
 >>> f = partial(isinstance, 1, float)
 >>> f()
False

> I have to wrap it in a lambda

Are you sure that partial will not work for you? Since partial is 
written in Python, you can grab and adjust the code to your needs. It 
amounts to adding default args after the fact by using a generic closure.

 > (or use .apply(), in the case of
> Elementwise).  This is not a huge deal for me, but it forces me to
> create wrappers for lots of functions (e.g. isinstance_(X, someclass))
> and/or have users wrap every such function they want to use in a
> symbolic expression.   Having to do this also bloats the code a lot;
> the github version of Elementwise is over 3,000 LoC at this point
> (including prodigious documentation, but still...).
>
> 2.  Python expects that certain functions, such as int(), str(), etc,
> will have a specific return type.  While in general I agree with this,

People expect that class constructors produce an instance of the class. 
It is surprising when one does otherwise ;-). Builtin classes like int, 
bool, and str are classes just like ones you write.

> it makes Elementwise somewhat inconsistent (and it will do the same to
> anything else that wants to work with symbolic expressions).
>
> I'm interested in fixing both issues. I believe both issues I've had
> could be solved by having a robust "symbolic object".  These objects
> would basically usable like ordinary objects, however upon any
> attribute access or other form of interaction, the object would
> basically short circuit the calling function, and return a symbolic
> object directly to the outer scope.  The symbolic object would behave
> like a generator function frozen at the point of attribute access, and
> upon send()-ing (or whatever method), it would behave exactly as if
> the values sent had been the ones passed in originally (ideally
> without consuming the generator).
>
> I have thought about ways to approximate this behavior python
> currently, and while I could hack something together using inspect to
> pull relevant info from the stack, then break out using a special
> exception (potentially passing a generator with state as an exception
> arg), this approach strikes me as VERY brittle, implementation
> dependent, ugly and difficult to work with.  Additionally, you would
> need to catch the special exception somewhere in the stack, so this
> trick wouldn't work on the first thing in an expression to be
> evaluated.
>
> As an aside, I'd like to solicit some feedback on the validation
> syntax I've been working on.  Currently, I have code that support
> things like:
>
> X = SymbolicObject()
>
> const = Constraints(X * 2 + 1>= 5, X % 2 != 0)
> const2 = Constraints(X[-1] == "h")
> const3 = Constraints(X[-1].upper() == "H")

Using a special class is a standard way to delay concrete execution.

>>>> print isinstance(3, const)
> True

A Contraints instance defines a set. 'const' is the set 'odd_ge_3'
It would look better if you used standard syntax and do the inclusion 
check in a __contains__ method.

 >>> 3 in odd_ge_3
True

>>>> print isinstance(2, const)
> False

2 in odd_ge_3
1 in odd_ge_3

>>>> print isinstance(1, const)
> False
>
>>>> print isinstance("bleh", const2)
> True
>
>>> print isinstance("bleh", const3)
> True

"bleh" in ends_in_h
"bleh" in ends_in_h_or_H

> Callables are supported as well, so if you wanted to do something like:
>
> Constraints(isinstance(X.attr, someclass), somefunc(X[-2].attr, args))
>
> You could approximate that with:
>
> Constraints(lambda x: isinstance(x.attr, someclass), lambda x:
> somefunc(x[-2].attr, args))
>
> As I mentioned in the first paragraph, Constraints is a metaclass,

This is your first mention, actually.

 > so your validations are checked using __instancecheck__.

But it is a fake check in that 3 is not really an instance of the class, 
which has no instances. It is hard for me to believe that you cannot put 
the same constraint data in an instance of Constraint as a class and the 
inclusion logic in __contains__. If you customize __instancecheck__ for 
each instance of Constraints, then write

   def__contains__(self, ob): return self._check(ob)

where _check does the same as your current __instancecheck__

Even if you *have* to make Constraints a metaclass, for other reasons, I 
believe you could still give it the same __contains__ method. A 
metaclass *is* a class, and if its class instances represent 
collections, inclusion in the colleciton should be tested in the 
standard way.

> I'm also
> considering having __init__ generate mock objects (for certain
> straight-forward cases, anyhow).

-- 
Terry Jan Reedy




More information about the Python-ideas mailing list