Python-ideas September 2011

python-ideas@python.org

79 participants
44 discussions

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 21 Feb '20

21 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

Tweaking closures and lexical scoping to include the function being defined
by Nick Coghlan 04 Oct '11

04 Oct '11

The problem of the 'default argument hack' and it's use for early binding and shared state in function definitions is one that has been bugging me for years. You could rightly say that the degree to which it irritates me is all out of proportion to the significance of the use case and frequency with which it arises, and I'd agree with you. That, at least in part, is what has made it such an interesting problem for me: the status quo is suboptimal and confusing (there's a reason the term 'default argument hack' gets thrown around, including by me), but most proposed solutions have involved fairly significant changes to the semantics and syntax of the language that cannot be justified by such a niche use case. The proposed cures (including my own suggestions) have all been worse than the disease. I finally have a possible answer I actually *like* ("nonlocal VAR from EXPR"), but it involves a somewhat novel way of thinking about closures and lexical scopes for it to make sense. This post is an attempt to explain that thought process. The history outlined here will be familiar to many folks on the list, but I hope to make the case that this approach actually simplifies and unifies a few aspects of the language rather than adding anything fundamentally new. The novel aspect lies in recognising and exposing to developers as a coherent feature something that is already implicit in the operation of the language as a whole. == Default Arguments == Default arguments have been a part of Python function definitions for a very long time (since the beginning, even?), so it makes sense to start with those. At function call time, if the relevant parameters are not supplied as arguments, they're populated on the frame object based on the values stored on the function object. Their behaviour is actually quite like a closure: they define shared state that is common to all invocations of the function. == Lexical Scoping == The second step in this journey is the original introduction of lexical scoping by PEP 227 back in Python 2.1 (or 2.2 without a __future__ statement). This changed Python from its original locals->globals->builtins lookup mechanism (still used in class scope to this day), to the closure semantics for nested functions that we're familiar with. However, at this stage, there was no ability to rebind names in outer scopes - they were read-only, so you needed to use other techniques (like 'boxing' in a list) to update immutable values. == Writing to Outer Scopes == PEP 3104 added the ability to write to outer scopes by using the 'nonlocal' statement to declare that a particular variable was not a local in the current frame, but rather a local in an outer frame which is alive at the time the inner function definition statement is executed. It expects the variable to already exist in an outer lexically nested scope and complains if it can't find one. == The "__class__" cell reference == The final entrant in this game, the "__class__" cell reference was added to the language by PEP 335 in order to implement the 3.x super() shorthand. For functions defined within a class body, this effectively lets the class definition play a role in lexical scoping, as the compiler and eval loop cooperate to give the function an indirect reference to the class being defined, even though the function definition completes first. == The Status Quo == If you go look up the definition of 'closure', you'll find that it doesn't actually say anything about nested functions. Instead, it will talk about 'free variables' in the algorithm definition without placing any restrictions on how those variables are later hooked up to the appropriate values. In current Python, ordinary named references can refer to one of 4 namespaces: - locals (stored on the currently executing frame object) - closure reference (stored in a cell object by the function that defined it, kept alive after the frame is recycled by references from still living inner functions that need it) - globals (stored on the module object) - builtins (also stored on a module object, specifically the one for the builtin namespace) PEP 335 also creates a closure reference for "__class__", but in a slightly unusual way. Whereas most targets for closure references are created by the code in the outer function when it runs [1], this closure reference is populated implicitly by the type machinery [2]. The important aspect from my point of view is that this implementation technique starts to break down Python's historical correlation between "function closure" and "lexically nested scope". == Conceptual Unification == The moment of clarity for me came when I realised that default arguments, lexically nested scopes and the new super() implementation can all be seen as just special cases of the broader concept of free variables and function closures. Lexically nested scopes have always been talked about in those terms, so that aspect shouldn't surprise anyone. The new super() implementation is also fairly obviously a closure, since it uses the closure machinery to work its magic. The only difference is in the way the value gets populated in the first place (i.e. by the type machinery rather than by the execution of an outer function). Due to history, default argument *values* aren't often thought of as closure references, but they really are anonymous closures. Instead of using cells, the references are stored in dedicated attributes that are known to the argument parsing machinery, but you could quite easily dispense with that and store everything as cells in the function closure (you wouldn't, since it would be a waste of time and energy, I'm just pointing out the conceptual equivalence. A *new* Python implementation, though, could choose to go down that path). After I had that realisation, the natural follow-up question seemed to be: if I wanted to explicitly declare a closure variable, and provide it with an initial value, without introducing a nested function purely for that purpose, how should I spell that? Well, I think PEP 3104 has already given us the answer: by declaring the variable name as explicitly 'nonlocal', but also providing an initial value so the compiler knows it is a *new* closure variable, rather than one from an outer lexically nested scope. This is a far more useful and meaningful addition than the trivial syntactic sugar mentioned in the PEP (but ultimately not implemented). The other question is what scope the initialisation operation should be executed in, and I think there, default arguments have the answer: in the containing scope, before the function has been defined. == Precise Syntax == By reusing 'nonlocal', we would make it clear that we're not adding a new concept to the language, but rather generalising an existing one (i.e. closure references) to provide additional flexibility in the way they're used. So I *really* want to use that keyword rather than adding a new one just for this task. However, I'm less certain about the spelling of the rest of the statement. There are at least a few possible alternative spellings: nonlocal VAR = EXPR # My initial suggestion nonlocal VAR from EXPR # Strongly indicates there's more than a simple assignment going on here nonlocal EXPR as VAR # Parser may struggle with this one Of the three, 'nonlocal VAR from EXPR' may be the best bet - it's easy for the compiler to parse, PEP 380 set the precedent for the 'from EXPR' clause to introduce a subexpression and 'nonlocal VAR = EXPR' may be too close to 'nonlocal VAR; VAR = EXPR'. Regards, Nick. [1] Some dis module details regarding the different kinds of name reference. Most notable for my point is the correspondence between the 'cell variable' in the outer function and the 'free variable' in the inner function: >>> def outer(): ... closure_ref = 1 ... def inner(): ... local_ref = 2 ... print(local_ref, closure_ref, global_ref, len) ... >>> global_ref = 3 >>> import dis >>> dis.show_code(outer) Name: outer Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS Constants: 0: None 1: 1 2: <code object inner at 0xee78b0, file "<stdin>", line 3> Variable names: 0: inner Cell variables: 0: closure_ref >>> dis.show_code(outer.__code__.co_consts[2]) Name: inner Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 5 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None 1: 2 Names: 0: print 1: global_ref 2: len Variable names: 0: local_ref Free variables: 0: closure_ref [2] Some dis module output to show that there's no corresponding '__class__' cell variable anywhere when the implicit closure entry is created by the new super() machinery. >>> def outer2(): ... class C: ... def inner(): ... print(__class__) ... >>> dis.show_code(outer2) Name: outer2 Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 3 Flags: OPTIMIZED, NEWLOCALS, NOFREE Constants: 0: None 1: <code object C at 0x1275c68, file "<stdin>", line 2> 2: 'C' Variable names: 0: C >>> dis.show_code(outer2.__code__.co_consts[1]) Name: C Filename: <stdin> Argument count: 1 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: NEWLOCALS Constants: 0: <code object inner at 0x1275608, file "<stdin>", line 3> Names: 0: __name__ 1: __module__ 2: inner Variable names: 0: __locals__ Cell variables: 0: __class__ >>> dis.show_code(outer2.__code__.co_consts[1].co_consts[0]) Name: inner Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 0 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None Names: 0: print Free variables: 0: __class__ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

33 213

Add peekline(), peeklines(n) and optional maxlines argument to readlines()
by John O'Connor 04 Oct '11

04 Oct '11

It seems there could be a cleaner way of reading the first n lines of a file and additionally not seeking past those lines (ie peek). This is obviously a trivial task for 1 line ie... f.readline() f.seek(0) but one that I think would make sense to add to the IO implementation, given that we already have readline, readlines, and peek I think peekline() or peeklines(n) is only a natural addition. The argument for doing so (in 3.3 of course), is primarily readability but also that the maintenance burden *seems* like it would be low. This addition would also be helpful and more concise where n > 1. I think readlines() should also take an optional argument for a max number of lines to read. It seems more common/helpful to me than 'hint' for max bytes. In n>1 case one could do... f.readlines(maxlines=10) or for the 'peek' case f.peeklines(10) I also didn't find any of the answers from http://stackoverflow.com/questions/1767513/read-first-n-lines-of-a-file-in-… to be very compelling. I am more than willing to propose a patch if the idea(s) are supported. - John

5 9

startsin ?
by Tarek Ziadé 03 Oct '11

03 Oct '11

Hey, not sure how people do this, or if I missed something obvious in the stdlib, but I often have this pattern: starts = ('a', 'b', 'c') somestring = 'acapulco' for start in starts: if somestring.startswith(start): print "yeah" So what about a startsin() method, that would iterate over a sequence: if somestring.startsin('a', 'b', 'c'): print "yeah" Implementing it in C should be faster as well same deal with .endswith I guess Cheers Tarek -- Tarek Ziadé | http://ziade.org

12 16

Proposal: add a calculator statistics module
by Steven D'Aprano 01 Oct '11

01 Oct '11

I propose adding a basic calculator statistics module to the standard library, similar to the sorts of functions you would get on a scientific calculator: mean (average) variance (population and sample) standard deviation (population and sample) correlation coefficient and similar. I am volunteering to provide, and support, this module, written in pure Python so other implementations will be able to use it. Simple calculator-style statistics seem to me to be a fairly obvious "battery" to be included, more useful in practice than some functions already available such as factorial and the hyperbolic functions. The lack of a standard solution leads people who need basic stats to roll their own. This seems seductively simple, as the basic stats formulae are quite simple. Unfortunately doing it *correctly* is much harder than it seems. Variance, in particular, is prone to serious inaccuracies. Here is the most obvious algorithm, using the so-called "computational formula for the variance": def variance(data): # σ2 = 1/n**2 * (n*Σ(x**2) - (Σx)**2) n = len(data) s1 = sum(x**2 for x in data) s2 = sum(data) return (n*s1 - s2**2)/(n*n) Many stats text books recommend this as the best way to calculate variance, advice which makes sense when you're talking about hand calculations of small numbers of moderate sized data, but not for floating point. It appears to work: >>> data = [1, 2, 4, 5, 8] >>> variance(data) # exact value = 6 6.0 but unfortunately it is numerically unstable. Shifting all the data points by a constant amount shouldn't change the variance, but it does: >>> data = [x+1e12 for x in data] >>> variance(data) 171798691.84 Even worse, variance should never be negative: >>> variance(data*100) -1266637395.197952 Note that using math.fsum instead of the built-in sum does not fix the numeric instability problem, and it adds the additional problem that it coerces the data points to float. (If you use Decimal, this may not be what you want.) Here is an example of published code which suffers from exactly this problem: https://bitbucket.org/larsyencken/simplestats/src/c42e048a6625/src/basic.py and here is an example on StackOverflow. Note the most popular answer given is to use the Computational Formula, which is the wrong answer. http://stackoverflow.com/questions/2341340/calculate-mean-and-variance-with… I would like to add a module to the standard library to solve these sorts of simple stats problems the right way, once and for all. Thoughts, comments, objections or words of encouragement are welcome. -- Steven

10 12

strtr? (was startsin ?
by INADA Naoki 01 Oct '11

01 Oct '11

I think `strtr`_ in php is also very useful when escaping something. _ strtr: http://jp.php.net/manual/en/function.strtr.php For example: .. code-block:: php php> = strtr("foo\\\"bar\\'baz\\\\", array("\\\\"=>"\\", '\\"'=>'"', "\\'"=>"'")); "foo\"bar'baz\\" .. code-block:: python In [1]: "foo\\\"bar\\'baz\\\\".replace('\\"', '"').replace("\\'", "'").replace('\\\\', '\\') Out[1]: 'foo"bar\'baz\\' In Python, lookup of 'replace' method occurs many times and temporary strings is created many times too. It makes Python slower than php. And replacing order may cause very common mistake. .. code-block:: python In [4]: "foo\\\"bar\\'baz\\\\'".replace('\\\\', '\\').replace('\\"', '"').replace("\\'", "'") Out[4]: 'foo"bar\'baz\'' When I wrote HandlerSocket_ client in pure Python. I use dirty hack for speed. http://bazaar.launchpad.net/~songofacandy/+junk/pyhandlersocket/view/head:/… I believe Pythonic means simple and efficient. My code is not Pythonic at all. .. _HandlerSocket: https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL On Sat, Oct 1, 2011 at 12:30 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > if somestring.startswith(start): > print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > print "yeah" > > Implementing it in C should be faster as well > > same deal with .endswith I guess > > Cheers > Tarek > > -- > Tarek Ziadé | http://ziade.org > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- INADA Naoki <songofacandy(a)gmail.com>

3 2

PEP 335 Revision 2 (Overloadable Boolean Operators)
by Greg Ewing 30 Sep '11

30 Sep '11

Here's a draft of an update to PEP 335. It includes a couple of fully worked and tested examples, plus discussion of some potential simplifications and ways to optimise the generated bytecode. ------------------------------------------------------------- PEP: 335 Title: Overloadable Boolean Operators Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg(a)cosc.canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Aug-2004 Python-Version: Unspecified Post-History: 05-Sep-2004 Abstract ======== This PEP proposes an extension to permit objects to define their own meanings for the boolean operators 'and', 'or' and 'not', and suggests an efficient strategy for implementation. A prototype of this implementation is available for download. Background ========== Python does not currently provide any '__xxx__' special methods corresponding to the 'and', 'or' and 'not' boolean operators. In the case of 'and' and 'or', the most likely reason is that these operators have short-circuiting semantics, i.e. the second operand is not evaluated if the result can be determined from the first operand. The usual technique of providing special methods for these operators therefore would not work. There is no such difficulty in the case of 'not', however, and it would be straightforward to provide a special method for this operator. The rest of this proposal will therefore concentrate mainly on providing a way to overload 'and' and 'or'. Motivation ========== There are many applications in which it is natural to provide custom meanings for Python operators, and in some of these, having boolean operators excluded from those able to be customised can be inconvenient. Examples include: 1. NumPy, in which almost all the operators are defined on arrays so as to perform the appropriate operation between corresponding elements, and return an array of the results. For consistency, one would expect a boolean operation between two arrays to return an array of booleans, but this is not currently possible. There is a precedent for an extension of this kind: comparison operators were originally restricted to returning boolean results, and rich comparisons were added so that comparisons of NumPy arrays could return arrays of booleans. 2. A symbolic algebra system, in which a Python expression is evaluated in an environment which results in it constructing a tree of objects corresponding to the structure of the expression. 3. A relational database interface, in which a Python expression is used to construct an SQL query. A workaround often suggested is to use the bitwise operators '&', '|' and '~' in place of 'and', 'or' and 'not', but this has some drawbacks. The precedence of these is different in relation to the other operators, and they may already be in use for other purposes (as in example 1). There is also the aesthetic consideration of forcing users to use something other than the most obvious syntax for what they are trying to express. This would be particularly acute in the case of example 3, considering that boolean operations are a staple of SQL queries. Rationale ========= The requirements for a successful solution to the problem of allowing boolean operators to be customised are: 1. In the default case (where there is no customisation), the existing short-circuiting semantics must be preserved. 2. There must not be any appreciable loss of speed in the default case. 3. Ideally, the customisation mechanism should allow the object to provide either short-circuiting or non-short-circuiting semantics, at its discretion. One obvious strategy, that has been previously suggested, is to pass into the special method the first argument and a function for evaluating the second argument. This would satisfy requirements 1 and 3, but not requirement 2, since it would incur the overhead of constructing a function object and possibly a Python function call on every boolean operation. Therefore, it will not be considered further here. The following section proposes a strategy that addresses all three requirements. A `prototype implementation`_ of this strategy is available for download. .. _prototype implementation: http://www.cosc.canterbury.ac.nz/~greg/python/obo//Python_OBO.tar.gz Specification ============= Special Methods --------------- At the Python level, objects may define the following special methods. =============== ================= ======================== Unary Binary, phase 1 Binary, phase 2 =============== ================= ======================== * __not__(self) * __and1__(self) * __and2__(self, other) * __or1__(self) * __or2__(self, other) * __rand2__(self, other) * __ror2__(self, other) =============== ================= ======================== The __not__ method, if defined, implements the 'not' operator. If it is not defined, or it returns NotImplemented, existing semantics are used. To permit short-circuiting, processing of the 'and' and 'or' operators is split into two phases. Phase 1 occurs after evaluation of the first operand but before the second. If the first operand defines the relevant phase 1 method, it is called with the first operand as argument. If that method can determine the result without needing the second operand, it returns the result, and further processing is skipped. If the phase 1 method determines that the second operand is needed, it returns the special value NeedOtherOperand. This triggers the evaluation of the second operand, and the calling of a relevant phase 2 method. During phase 2, the __and2__/__rand2__ and __or2__/__ror2__ method pairs work as for other binary operators. Processing falls back to existing semantics if at any stage a relevant special method is not found or returns NotImplemented. As a special case, if the first operand defines a phase 2 method but no corresponding phase 1 method, the second operand is always evaluated and the phase 2 method called. This allows an object which does not want short-circuiting semantics to simply implement the phase 2 methods and ignore phase 1. Bytecodes --------- The patch adds four new bytecodes, LOGICAL_AND_1, LOGICAL_AND_2, LOGICAL_OR_1 and LOGICAL_OR_2. As an example of their use, the bytecode generated for an 'and' expression looks like this:: . . . evaluate first operand LOGICAL_AND_1 L evaluate second operand LOGICAL_AND_2 L: . . . The LOGICAL_AND_1 bytecode performs phase 1 processing. If it determines that the second operand is needed, it leaves the first operand on the stack and continues with the following code. Otherwise it pops the first operand, pushes the result and branches to L. The LOGICAL_AND_2 bytecode performs phase 2 processing, popping both operands and pushing the result. Type Slots ---------- At the C level, the new special methods are manifested as five new slots in the type object. In the patch, they are added to the tp_as_number substructure, since this allows making use of some existing code for dealing with unary and binary operators. Their existence is signalled by a new type flag, Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD. The new type slots are:: unaryfunc nb_logical_not; unaryfunc nb_logical_and_1; unaryfunc nb_logical_or_1; binaryfunc nb_logical_and_2; binaryfunc nb_logical_or_2; Python/C API Functions ---------------------- There are also five new Python/C API functions corresponding to the new operations:: PyObject *PyObject_LogicalNot(PyObject *); PyObject *PyObject_LogicalAnd1(PyObject *); PyObject *PyObject_LogicalOr1(PyObject *); PyObject *PyObject_LogicalAnd2(PyObject *, PyObject *); PyObject *PyObject_LogicalOr2(PyObject *, PyObject *); Alternatives and Optimisations ============================== This section discusses some possible variations on the proposal, and ways in which the bytecode sequences generated for boolean expressions could be optimised. Reduced special method set -------------------------- For completeness, the full version of this proposal includes a mechanism for types to define their own customised short-circuiting behaviour. However, the full mechanism is not needed to address the main use cases put forward here, and it would be possible to define a simplified version that only includes the phase 2 methods. There would then only be 5 new special methods (__and2__, __rand2__, __or2__, __ror2__, __not__) with 3 associated type slots and 3 API functions. This simplified version could be expanded to the full version later if desired. Additional bytecodes -------------------- As defined here, the bytecode sequence for code that branches on the result of a boolean expression would be slightly longer than it currently is. For example, in Python 2.7, :: if a and b: statement1 else: statement2 generates LOAD_GLOBAL a POP_JUMP_IF_FALSE false_branch LOAD_GLOBAL b POP_JUMP_IF_FALSE false_branch <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: Under this proposal as described so far, it would become something like :: LOAD_GLOBAL a LOGICAL_AND_1 test LOAD_GLOBAL b LOGICAL_AND_2 test: POP_JUMP_IF_FALSE false_branch <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: This involves executing one extra bytecode in the short-circuiting case and two extra bytecodes in the non-short-circuiting case. However, by introducing extra bytecodes that combine the logical operations with testing and branching on the result, it can be reduced to the same number of bytecodes as the original: :: LOAD_GLOBAL a AND1_JUMP true_branch, false_branch LOAD_GLOBAL b AND2_JUMP_IF_FALSE false_branch true_branch: <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: Here, AND1_JUMP performs phase 1 processing as above, and then examines the result. If there is a result, it is popped from the stack, its truth value is tested and a branch taken to one of two locations. Otherwise, the first operand is left on the stack and execution continues to the next bytecode. The AND2_JUMP_IF_FALSE bytecode performs phase 2 processing, pops the result and branches if it tests false For the 'or' operator, there would be corresponding OR1_JUMP and OR2_JUMP_IF_TRUE bytecodes. If the simplified version without phase 1 methods is used, then early exiting can only occur if the first operand is false for 'and' and true for 'or'. Consequently, the two-target AND1_JUMP and OR1_JUMP bytecodes can be replaced with AND1_JUMP_IF_FALSE and OR1_JUMP_IF_TRUE, these being ordinary branch instructions with only one target. Optimisation of 'not' --------------------- Recent versions of Python implement a simple optimisation in which branching on a negated boolean expression is implemented by reversing the sense of the branch, saving a UNARY_NOT opcode. Taking a strict view, this optimisation should no longer be performed, because the 'not' operator may be overridden to produce quite different results from usual. However, in typical use cases, it is not envisaged that expressions involving customised boolean operations will be used for branching -- it is much more likely that the result will be used in some other way. Therefore, it would probably do little harm to specify that the compiler is allowed to use the laws of boolean algebra to simplify any expression that appears directly in a boolean context. If this is inconvenient, the result can always be assigned to a temporary name first. This would allow the existing 'not' optimisation to remain, and would permit future extensions of it such as using De Morgan's laws to extend it deeper into the expression. Usage Examples ============== Example 1: NumPy Arrays ----------------------- :: #----------------------------------------------------------------- # # This example creates a subclass of numpy array to which # 'and', 'or' and 'not' can be applied, producing an array # of booleans. # #----------------------------------------------------------------- from numpy import array, ndarray class BArray(ndarray): def __str__(self): return "barray(%s)" % ndarray.__str__(self) def __and2__(self, other): return (self & other) def __or2__(self, other): return (self & other) def __not__(self): return (self == 0) def barray(*args, **kwds): return array(*args, **kwds).view(type = BArray) a0 = barray([0, 1, 2, 4]) a1 = barray([1, 2, 3, 4]) a2 = barray([5, 6, 3, 4]) a3 = barray([5, 1, 2, 4]) print "a0:", a0 print "a1:", a1 print "a2:", a2 print "a3:", a3 print "not a0:", not a0 print "a0 == a1 and a2 == a3:", a0 == a1 and a2 == a3 print "a0 == a1 or a2 == a3:", a0 == a1 or a2 == a3 Example 1 Output --------------- :: a0: barray([0 1 2 4]) a1: barray([1 2 3 4]) a2: barray([5 6 3 4]) a3: barray([5 1 2 4]) not a0: barray([ True False False False]) a0 == a1 and a2 == a3: barray([False False False True]) a0 == a1 or a2 == a3: barray([False False False True]) Example 2: Database Queries --------------------------- :: #----------------------------------------------------------------- # # This example demonstrates the creation of a DSL for database # queries allowing 'and' and 'or' operators to be used to # formulate the query. # #----------------------------------------------------------------- class SQLNode(object): def __and2__(self, other): return SQLBinop("and", self, other) def __rand2__(self, other): return SQLBinop("and", other, self) def __eq__(self, other): return SQLBinop("=", self, other) class Table(SQLNode): def __init__(self, name): self.__tablename__ = name def __getattr__(self, name): return SQLAttr(self, name) def __sql__(self): return self.__tablename__ class SQLBinop(SQLNode): def __init__(self, op, opnd1, opnd2): self.op = op.upper() self.opnd1 = opnd1 self.opnd2 = opnd2 def __sql__(self): return "(%s %s %s)" % (sql(self.opnd1), self.op, sql(self.opnd2)) class SQLAttr(SQLNode): def __init__(self, table, name): self.table = table self.name = name def __sql__(self): return "%s.%s" % (sql(self.table), self.name) class SQLSelect(SQLNode): def __init__(self, targets): self.targets = targets self.where_clause = None def where(self, expr): self.where_clause = expr return self def __sql__(self): result = "SELECT %s" % ", ".join([sql(target) for target in self.targets]) if self.where_clause: result = "%s WHERE %s" % (result, sql(self.where_clause)) return result def sql(expr): if isinstance(expr, SQLNode): return expr.__sql__() elif isinstance(expr, str): return "'%s'" % expr.replace("'", "''") else: return str(expr) def select(*targets): return SQLSelect(targets) #-------------------------------------------------------------------------------- dishes = Table("dishes") customers = Table("customers") orders = Table("orders") query = select(customers.name, dishes.price, orders.amount).where( customers.cust_id == orders.cust_id and orders.dish_id == dishes.dish_id and dishes.name == "Spam, Eggs, Sausages and Spam") print repr(query) print sql(query) Example 2 Output ---------------- :: <__main__.SQLSelect object at 0x1cc830> SELECT customers.name, dishes.price, orders.amount WHERE (((customers.cust_id = orders.cust_id) AND (orders.dish_id = dishes.dish_id)) AND (dishes.name = 'Spam, Eggs, Sausages and Spam')) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2011