Mailman 3 September 2011 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

Tweaking closures and lexical scoping to include the function being defined
by Nick Coghlan 05 Oct '11

05 Oct '11

The problem of the 'default argument hack' and it's use for early binding and shared state in function definitions is one that has been bugging me for years. You could rightly say that the degree to which it irritates me is all out of proportion to the significance of the use case and frequency with which it arises, and I'd agree with you. That, at least in part, is what has made it such an interesting problem for me: the status quo is suboptimal and confusing (there's a reason the term 'default argument hack' gets thrown around, including by me), but most proposed solutions have involved fairly significant changes to the semantics and syntax of the language that cannot be justified by such a niche use case. The proposed cures (including my own suggestions) have all been worse than the disease. I finally have a possible answer I actually *like* ("nonlocal VAR from EXPR"), but it involves a somewhat novel way of thinking about closures and lexical scopes for it to make sense. This post is an attempt to explain that thought process. The history outlined here will be familiar to many folks on the list, but I hope to make the case that this approach actually simplifies and unifies a few aspects of the language rather than adding anything fundamentally new. The novel aspect lies in recognising and exposing to developers as a coherent feature something that is already implicit in the operation of the language as a whole. == Default Arguments == Default arguments have been a part of Python function definitions for a very long time (since the beginning, even?), so it makes sense to start with those. At function call time, if the relevant parameters are not supplied as arguments, they're populated on the frame object based on the values stored on the function object. Their behaviour is actually quite like a closure: they define shared state that is common to all invocations of the function. == Lexical Scoping == The second step in this journey is the original introduction of lexical scoping by PEP 227 back in Python 2.1 (or 2.2 without a __future__ statement). This changed Python from its original locals->globals->builtins lookup mechanism (still used in class scope to this day), to the closure semantics for nested functions that we're familiar with. However, at this stage, there was no ability to rebind names in outer scopes - they were read-only, so you needed to use other techniques (like 'boxing' in a list) to update immutable values. == Writing to Outer Scopes == PEP 3104 added the ability to write to outer scopes by using the 'nonlocal' statement to declare that a particular variable was not a local in the current frame, but rather a local in an outer frame which is alive at the time the inner function definition statement is executed. It expects the variable to already exist in an outer lexically nested scope and complains if it can't find one. == The "__class__" cell reference == The final entrant in this game, the "__class__" cell reference was added to the language by PEP 335 in order to implement the 3.x super() shorthand. For functions defined within a class body, this effectively lets the class definition play a role in lexical scoping, as the compiler and eval loop cooperate to give the function an indirect reference to the class being defined, even though the function definition completes first. == The Status Quo == If you go look up the definition of 'closure', you'll find that it doesn't actually say anything about nested functions. Instead, it will talk about 'free variables' in the algorithm definition without placing any restrictions on how those variables are later hooked up to the appropriate values. In current Python, ordinary named references can refer to one of 4 namespaces: - locals (stored on the currently executing frame object) - closure reference (stored in a cell object by the function that defined it, kept alive after the frame is recycled by references from still living inner functions that need it) - globals (stored on the module object) - builtins (also stored on a module object, specifically the one for the builtin namespace) PEP 335 also creates a closure reference for "__class__", but in a slightly unusual way. Whereas most targets for closure references are created by the code in the outer function when it runs [1], this closure reference is populated implicitly by the type machinery [2]. The important aspect from my point of view is that this implementation technique starts to break down Python's historical correlation between "function closure" and "lexically nested scope". == Conceptual Unification == The moment of clarity for me came when I realised that default arguments, lexically nested scopes and the new super() implementation can all be seen as just special cases of the broader concept of free variables and function closures. Lexically nested scopes have always been talked about in those terms, so that aspect shouldn't surprise anyone. The new super() implementation is also fairly obviously a closure, since it uses the closure machinery to work its magic. The only difference is in the way the value gets populated in the first place (i.e. by the type machinery rather than by the execution of an outer function). Due to history, default argument *values* aren't often thought of as closure references, but they really are anonymous closures. Instead of using cells, the references are stored in dedicated attributes that are known to the argument parsing machinery, but you could quite easily dispense with that and store everything as cells in the function closure (you wouldn't, since it would be a waste of time and energy, I'm just pointing out the conceptual equivalence. A *new* Python implementation, though, could choose to go down that path). After I had that realisation, the natural follow-up question seemed to be: if I wanted to explicitly declare a closure variable, and provide it with an initial value, without introducing a nested function purely for that purpose, how should I spell that? Well, I think PEP 3104 has already given us the answer: by declaring the variable name as explicitly 'nonlocal', but also providing an initial value so the compiler knows it is a *new* closure variable, rather than one from an outer lexically nested scope. This is a far more useful and meaningful addition than the trivial syntactic sugar mentioned in the PEP (but ultimately not implemented). The other question is what scope the initialisation operation should be executed in, and I think there, default arguments have the answer: in the containing scope, before the function has been defined. == Precise Syntax == By reusing 'nonlocal', we would make it clear that we're not adding a new concept to the language, but rather generalising an existing one (i.e. closure references) to provide additional flexibility in the way they're used. So I *really* want to use that keyword rather than adding a new one just for this task. However, I'm less certain about the spelling of the rest of the statement. There are at least a few possible alternative spellings: nonlocal VAR = EXPR # My initial suggestion nonlocal VAR from EXPR # Strongly indicates there's more than a simple assignment going on here nonlocal EXPR as VAR # Parser may struggle with this one Of the three, 'nonlocal VAR from EXPR' may be the best bet - it's easy for the compiler to parse, PEP 380 set the precedent for the 'from EXPR' clause to introduce a subexpression and 'nonlocal VAR = EXPR' may be too close to 'nonlocal VAR; VAR = EXPR'. Regards, Nick. [1] Some dis module details regarding the different kinds of name reference. Most notable for my point is the correspondence between the 'cell variable' in the outer function and the 'free variable' in the inner function: >>> def outer(): ... closure_ref = 1 ... def inner(): ... local_ref = 2 ... print(local_ref, closure_ref, global_ref, len) ... >>> global_ref = 3 >>> import dis >>> dis.show_code(outer) Name: outer Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS Constants: 0: None 1: 1 2: <code object inner at 0xee78b0, file "<stdin>", line 3> Variable names: 0: inner Cell variables: 0: closure_ref >>> dis.show_code(outer.__code__.co_consts[2]) Name: inner Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 5 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None 1: 2 Names: 0: print 1: global_ref 2: len Variable names: 0: local_ref Free variables: 0: closure_ref [2] Some dis module output to show that there's no corresponding '__class__' cell variable anywhere when the implicit closure entry is created by the new super() machinery. >>> def outer2(): ... class C: ... def inner(): ... print(__class__) ... >>> dis.show_code(outer2) Name: outer2 Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 3 Flags: OPTIMIZED, NEWLOCALS, NOFREE Constants: 0: None 1: <code object C at 0x1275c68, file "<stdin>", line 2> 2: 'C' Variable names: 0: C >>> dis.show_code(outer2.__code__.co_consts[1]) Name: C Filename: <stdin> Argument count: 1 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: NEWLOCALS Constants: 0: <code object inner at 0x1275608, file "<stdin>", line 3> Names: 0: __name__ 1: __module__ 2: inner Variable names: 0: __locals__ Cell variables: 0: __class__ >>> dis.show_code(outer2.__code__.co_consts[1].co_consts[0]) Name: inner Filename: <stdin> Argument count: 0 Kw-only arguments: 0 Number of locals: 0 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None Names: 0: print Free variables: 0: __class__ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

33 213

Add peekline(), peeklines(n) and optional maxlines argument to readlines()
by John O'Connor 04 Oct '11

04 Oct '11

It seems there could be a cleaner way of reading the first n lines of a file and additionally not seeking past those lines (ie peek). This is obviously a trivial task for 1 line ie... f.readline() f.seek(0) but one that I think would make sense to add to the IO implementation, given that we already have readline, readlines, and peek I think peekline() or peeklines(n) is only a natural addition. The argument for doing so (in 3.3 of course), is primarily readability but also that the maintenance burden *seems* like it would be low. This addition would also be helpful and more concise where n > 1. I think readlines() should also take an optional argument for a max number of lines to read. It seems more common/helpful to me than 'hint' for max bytes. In n>1 case one could do... f.readlines(maxlines=10) or for the 'peek' case f.peeklines(10) I also didn't find any of the answers from http://stackoverflow.com/questions/1767513/read-first-n-lines-of-a-file-in-… to be very compelling. I am more than willing to propose a patch if the idea(s) are supported. - John

5 9

startsin ?
by Tarek Ziadé 03 Oct '11

03 Oct '11

Hey, not sure how people do this, or if I missed something obvious in the stdlib, but I often have this pattern: starts = ('a', 'b', 'c') somestring = 'acapulco' for start in starts: if somestring.startswith(start): print "yeah" So what about a startsin() method, that would iterate over a sequence: if somestring.startsin('a', 'b', 'c'): print "yeah" Implementing it in C should be faster as well same deal with .endswith I guess Cheers Tarek -- Tarek Ziadé | http://ziade.org

12 16

Proposal: add a calculator statistics module
by Steven D'Aprano 01 Oct '11

01 Oct '11

I propose adding a basic calculator statistics module to the standard library, similar to the sorts of functions you would get on a scientific calculator: mean (average) variance (population and sample) standard deviation (population and sample) correlation coefficient and similar. I am volunteering to provide, and support, this module, written in pure Python so other implementations will be able to use it. Simple calculator-style statistics seem to me to be a fairly obvious "battery" to be included, more useful in practice than some functions already available such as factorial and the hyperbolic functions. The lack of a standard solution leads people who need basic stats to roll their own. This seems seductively simple, as the basic stats formulae are quite simple. Unfortunately doing it *correctly* is much harder than it seems. Variance, in particular, is prone to serious inaccuracies. Here is the most obvious algorithm, using the so-called "computational formula for the variance": def variance(data): # σ2 = 1/n**2 * (n*Σ(x**2) - (Σx)**2) n = len(data) s1 = sum(x**2 for x in data) s2 = sum(data) return (n*s1 - s2**2)/(n*n) Many stats text books recommend this as the best way to calculate variance, advice which makes sense when you're talking about hand calculations of small numbers of moderate sized data, but not for floating point. It appears to work: >>> data = [1, 2, 4, 5, 8] >>> variance(data) # exact value = 6 6.0 but unfortunately it is numerically unstable. Shifting all the data points by a constant amount shouldn't change the variance, but it does: >>> data = [x+1e12 for x in data] >>> variance(data) 171798691.84 Even worse, variance should never be negative: >>> variance(data*100) -1266637395.197952 Note that using math.fsum instead of the built-in sum does not fix the numeric instability problem, and it adds the additional problem that it coerces the data points to float. (If you use Decimal, this may not be what you want.) Here is an example of published code which suffers from exactly this problem: https://bitbucket.org/larsyencken/simplestats/src/c42e048a6625/src/basic.py and here is an example on StackOverflow. Note the most popular answer given is to use the Computational Formula, which is the wrong answer. http://stackoverflow.com/questions/2341340/calculate-mean-and-variance-with… I would like to add a module to the standard library to solve these sorts of simple stats problems the right way, once and for all. Thoughts, comments, objections or words of encouragement are welcome. -- Steven

10 12

strtr? (was startsin ?
by INADA Naoki 01 Oct '11

01 Oct '11

I think `strtr`_ in php is also very useful when escaping something. _ strtr: http://jp.php.net/manual/en/function.strtr.php For example: .. code-block:: php php> = strtr("foo\\\"bar\\'baz\\\\", array("\\\\"=>"\\", '\\"'=>'"', "\\'"=>"'")); "foo\"bar'baz\\" .. code-block:: python In [1]: "foo\\\"bar\\'baz\\\\".replace('\\"', '"').replace("\\'", "'").replace('\\\\', '\\') Out[1]: 'foo"bar\'baz\\' In Python, lookup of 'replace' method occurs many times and temporary strings is created many times too. It makes Python slower than php. And replacing order may cause very common mistake. .. code-block:: python In [4]: "foo\\\"bar\\'baz\\\\'".replace('\\\\', '\\').replace('\\"', '"').replace("\\'", "'") Out[4]: 'foo"bar\'baz\'' When I wrote HandlerSocket_ client in pure Python. I use dirty hack for speed. http://bazaar.launchpad.net/~songofacandy/+junk/pyhandlersocket/view/head:/… I believe Pythonic means simple and efficient. My code is not Pythonic at all. .. _HandlerSocket: https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL On Sat, Oct 1, 2011 at 12:30 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > if somestring.startswith(start): > print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > print "yeah" > > Implementing it in C should be faster as well > > same deal with .endswith I guess > > Cheers > Tarek > > -- > Tarek Ziadé | http://ziade.org > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- INADA Naoki <songofacandy(a)gmail.com>

3 2

PEP 335 Revision 2 (Overloadable Boolean Operators)
by Greg Ewing 01 Oct '11

01 Oct '11

Here's a draft of an update to PEP 335. It includes a couple of fully worked and tested examples, plus discussion of some potential simplifications and ways to optimise the generated bytecode. ------------------------------------------------------------- PEP: 335 Title: Overloadable Boolean Operators Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg(a)cosc.canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Aug-2004 Python-Version: Unspecified Post-History: 05-Sep-2004 Abstract ======== This PEP proposes an extension to permit objects to define their own meanings for the boolean operators 'and', 'or' and 'not', and suggests an efficient strategy for implementation. A prototype of this implementation is available for download. Background ========== Python does not currently provide any '__xxx__' special methods corresponding to the 'and', 'or' and 'not' boolean operators. In the case of 'and' and 'or', the most likely reason is that these operators have short-circuiting semantics, i.e. the second operand is not evaluated if the result can be determined from the first operand. The usual technique of providing special methods for these operators therefore would not work. There is no such difficulty in the case of 'not', however, and it would be straightforward to provide a special method for this operator. The rest of this proposal will therefore concentrate mainly on providing a way to overload 'and' and 'or'. Motivation ========== There are many applications in which it is natural to provide custom meanings for Python operators, and in some of these, having boolean operators excluded from those able to be customised can be inconvenient. Examples include: 1. NumPy, in which almost all the operators are defined on arrays so as to perform the appropriate operation between corresponding elements, and return an array of the results. For consistency, one would expect a boolean operation between two arrays to return an array of booleans, but this is not currently possible. There is a precedent for an extension of this kind: comparison operators were originally restricted to returning boolean results, and rich comparisons were added so that comparisons of NumPy arrays could return arrays of booleans. 2. A symbolic algebra system, in which a Python expression is evaluated in an environment which results in it constructing a tree of objects corresponding to the structure of the expression. 3. A relational database interface, in which a Python expression is used to construct an SQL query. A workaround often suggested is to use the bitwise operators '&', '|' and '~' in place of 'and', 'or' and 'not', but this has some drawbacks. The precedence of these is different in relation to the other operators, and they may already be in use for other purposes (as in example 1). There is also the aesthetic consideration of forcing users to use something other than the most obvious syntax for what they are trying to express. This would be particularly acute in the case of example 3, considering that boolean operations are a staple of SQL queries. Rationale ========= The requirements for a successful solution to the problem of allowing boolean operators to be customised are: 1. In the default case (where there is no customisation), the existing short-circuiting semantics must be preserved. 2. There must not be any appreciable loss of speed in the default case. 3. Ideally, the customisation mechanism should allow the object to provide either short-circuiting or non-short-circuiting semantics, at its discretion. One obvious strategy, that has been previously suggested, is to pass into the special method the first argument and a function for evaluating the second argument. This would satisfy requirements 1 and 3, but not requirement 2, since it would incur the overhead of constructing a function object and possibly a Python function call on every boolean operation. Therefore, it will not be considered further here. The following section proposes a strategy that addresses all three requirements. A `prototype implementation`_ of this strategy is available for download. .. _prototype implementation: http://www.cosc.canterbury.ac.nz/~greg/python/obo//Python_OBO.tar.gz Specification ============= Special Methods --------------- At the Python level, objects may define the following special methods. =============== ================= ======================== Unary Binary, phase 1 Binary, phase 2 =============== ================= ======================== * __not__(self) * __and1__(self) * __and2__(self, other) * __or1__(self) * __or2__(self, other) * __rand2__(self, other) * __ror2__(self, other) =============== ================= ======================== The __not__ method, if defined, implements the 'not' operator. If it is not defined, or it returns NotImplemented, existing semantics are used. To permit short-circuiting, processing of the 'and' and 'or' operators is split into two phases. Phase 1 occurs after evaluation of the first operand but before the second. If the first operand defines the relevant phase 1 method, it is called with the first operand as argument. If that method can determine the result without needing the second operand, it returns the result, and further processing is skipped. If the phase 1 method determines that the second operand is needed, it returns the special value NeedOtherOperand. This triggers the evaluation of the second operand, and the calling of a relevant phase 2 method. During phase 2, the __and2__/__rand2__ and __or2__/__ror2__ method pairs work as for other binary operators. Processing falls back to existing semantics if at any stage a relevant special method is not found or returns NotImplemented. As a special case, if the first operand defines a phase 2 method but no corresponding phase 1 method, the second operand is always evaluated and the phase 2 method called. This allows an object which does not want short-circuiting semantics to simply implement the phase 2 methods and ignore phase 1. Bytecodes --------- The patch adds four new bytecodes, LOGICAL_AND_1, LOGICAL_AND_2, LOGICAL_OR_1 and LOGICAL_OR_2. As an example of their use, the bytecode generated for an 'and' expression looks like this:: . . . evaluate first operand LOGICAL_AND_1 L evaluate second operand LOGICAL_AND_2 L: . . . The LOGICAL_AND_1 bytecode performs phase 1 processing. If it determines that the second operand is needed, it leaves the first operand on the stack and continues with the following code. Otherwise it pops the first operand, pushes the result and branches to L. The LOGICAL_AND_2 bytecode performs phase 2 processing, popping both operands and pushing the result. Type Slots ---------- At the C level, the new special methods are manifested as five new slots in the type object. In the patch, they are added to the tp_as_number substructure, since this allows making use of some existing code for dealing with unary and binary operators. Their existence is signalled by a new type flag, Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD. The new type slots are:: unaryfunc nb_logical_not; unaryfunc nb_logical_and_1; unaryfunc nb_logical_or_1; binaryfunc nb_logical_and_2; binaryfunc nb_logical_or_2; Python/C API Functions ---------------------- There are also five new Python/C API functions corresponding to the new operations:: PyObject *PyObject_LogicalNot(PyObject *); PyObject *PyObject_LogicalAnd1(PyObject *); PyObject *PyObject_LogicalOr1(PyObject *); PyObject *PyObject_LogicalAnd2(PyObject *, PyObject *); PyObject *PyObject_LogicalOr2(PyObject *, PyObject *); Alternatives and Optimisations ============================== This section discusses some possible variations on the proposal, and ways in which the bytecode sequences generated for boolean expressions could be optimised. Reduced special method set -------------------------- For completeness, the full version of this proposal includes a mechanism for types to define their own customised short-circuiting behaviour. However, the full mechanism is not needed to address the main use cases put forward here, and it would be possible to define a simplified version that only includes the phase 2 methods. There would then only be 5 new special methods (__and2__, __rand2__, __or2__, __ror2__, __not__) with 3 associated type slots and 3 API functions. This simplified version could be expanded to the full version later if desired. Additional bytecodes -------------------- As defined here, the bytecode sequence for code that branches on the result of a boolean expression would be slightly longer than it currently is. For example, in Python 2.7, :: if a and b: statement1 else: statement2 generates LOAD_GLOBAL a POP_JUMP_IF_FALSE false_branch LOAD_GLOBAL b POP_JUMP_IF_FALSE false_branch <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: Under this proposal as described so far, it would become something like :: LOAD_GLOBAL a LOGICAL_AND_1 test LOAD_GLOBAL b LOGICAL_AND_2 test: POP_JUMP_IF_FALSE false_branch <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: This involves executing one extra bytecode in the short-circuiting case and two extra bytecodes in the non-short-circuiting case. However, by introducing extra bytecodes that combine the logical operations with testing and branching on the result, it can be reduced to the same number of bytecodes as the original: :: LOAD_GLOBAL a AND1_JUMP true_branch, false_branch LOAD_GLOBAL b AND2_JUMP_IF_FALSE false_branch true_branch: <code for statement1> JUMP_FORWARD end_branch false_branch: <code for statement2> end_branch: Here, AND1_JUMP performs phase 1 processing as above, and then examines the result. If there is a result, it is popped from the stack, its truth value is tested and a branch taken to one of two locations. Otherwise, the first operand is left on the stack and execution continues to the next bytecode. The AND2_JUMP_IF_FALSE bytecode performs phase 2 processing, pops the result and branches if it tests false For the 'or' operator, there would be corresponding OR1_JUMP and OR2_JUMP_IF_TRUE bytecodes. If the simplified version without phase 1 methods is used, then early exiting can only occur if the first operand is false for 'and' and true for 'or'. Consequently, the two-target AND1_JUMP and OR1_JUMP bytecodes can be replaced with AND1_JUMP_IF_FALSE and OR1_JUMP_IF_TRUE, these being ordinary branch instructions with only one target. Optimisation of 'not' --------------------- Recent versions of Python implement a simple optimisation in which branching on a negated boolean expression is implemented by reversing the sense of the branch, saving a UNARY_NOT opcode. Taking a strict view, this optimisation should no longer be performed, because the 'not' operator may be overridden to produce quite different results from usual. However, in typical use cases, it is not envisaged that expressions involving customised boolean operations will be used for branching -- it is much more likely that the result will be used in some other way. Therefore, it would probably do little harm to specify that the compiler is allowed to use the laws of boolean algebra to simplify any expression that appears directly in a boolean context. If this is inconvenient, the result can always be assigned to a temporary name first. This would allow the existing 'not' optimisation to remain, and would permit future extensions of it such as using De Morgan's laws to extend it deeper into the expression. Usage Examples ============== Example 1: NumPy Arrays ----------------------- :: #----------------------------------------------------------------- # # This example creates a subclass of numpy array to which # 'and', 'or' and 'not' can be applied, producing an array # of booleans. # #----------------------------------------------------------------- from numpy import array, ndarray class BArray(ndarray): def __str__(self): return "barray(%s)" % ndarray.__str__(self) def __and2__(self, other): return (self & other) def __or2__(self, other): return (self & other) def __not__(self): return (self == 0) def barray(*args, **kwds): return array(*args, **kwds).view(type = BArray) a0 = barray([0, 1, 2, 4]) a1 = barray([1, 2, 3, 4]) a2 = barray([5, 6, 3, 4]) a3 = barray([5, 1, 2, 4]) print "a0:", a0 print "a1:", a1 print "a2:", a2 print "a3:", a3 print "not a0:", not a0 print "a0 == a1 and a2 == a3:", a0 == a1 and a2 == a3 print "a0 == a1 or a2 == a3:", a0 == a1 or a2 == a3 Example 1 Output --------------- :: a0: barray([0 1 2 4]) a1: barray([1 2 3 4]) a2: barray([5 6 3 4]) a3: barray([5 1 2 4]) not a0: barray([ True False False False]) a0 == a1 and a2 == a3: barray([False False False True]) a0 == a1 or a2 == a3: barray([False False False True]) Example 2: Database Queries --------------------------- :: #----------------------------------------------------------------- # # This example demonstrates the creation of a DSL for database # queries allowing 'and' and 'or' operators to be used to # formulate the query. # #----------------------------------------------------------------- class SQLNode(object): def __and2__(self, other): return SQLBinop("and", self, other) def __rand2__(self, other): return SQLBinop("and", other, self) def __eq__(self, other): return SQLBinop("=", self, other) class Table(SQLNode): def __init__(self, name): self.__tablename__ = name def __getattr__(self, name): return SQLAttr(self, name) def __sql__(self): return self.__tablename__ class SQLBinop(SQLNode): def __init__(self, op, opnd1, opnd2): self.op = op.upper() self.opnd1 = opnd1 self.opnd2 = opnd2 def __sql__(self): return "(%s %s %s)" % (sql(self.opnd1), self.op, sql(self.opnd2)) class SQLAttr(SQLNode): def __init__(self, table, name): self.table = table self.name = name def __sql__(self): return "%s.%s" % (sql(self.table), self.name) class SQLSelect(SQLNode): def __init__(self, targets): self.targets = targets self.where_clause = None def where(self, expr): self.where_clause = expr return self def __sql__(self): result = "SELECT %s" % ", ".join([sql(target) for target in self.targets]) if self.where_clause: result = "%s WHERE %s" % (result, sql(self.where_clause)) return result def sql(expr): if isinstance(expr, SQLNode): return expr.__sql__() elif isinstance(expr, str): return "'%s'" % expr.replace("'", "''") else: return str(expr) def select(*targets): return SQLSelect(targets) #-------------------------------------------------------------------------------- dishes = Table("dishes") customers = Table("customers") orders = Table("orders") query = select(customers.name, dishes.price, orders.amount).where( customers.cust_id == orders.cust_id and orders.dish_id == dishes.dish_id and dishes.name == "Spam, Eggs, Sausages and Spam") print repr(query) print sql(query) Example 2 Output ---------------- :: <__main__.SQLSelect object at 0x1cc830> SELECT customers.name, dishes.price, orders.amount WHERE (((customers.cust_id = orders.cust_id) AND (orders.dish_id = dishes.dish_id)) AND (dishes.name = 'Spam, Eggs, Sausages and Spam')) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

2 1

Proposed additions to inspect module for function and generator state introspection
by Nick Coghlan 30 Sep '11

30 Sep '11

Based on the testability comments in the closure threads, I created http://bugs.python.org/issue13062 to propose two new introspection functions: inspect.getclosure(func) Returns a dictionary mapping closure references from the supplied function to their current values. inspect.getgeneratorlocals(generator) Returns the same result as would be reported by calling locals() in the generator's frame of execution The former would just involve syncing up the names on the code object with the cell references on the function object, while the latter would be equivalent to doing generator.gi_frame.f_locals with some nice error checking for when the generator's frame is already gone (or the supplied object isn't a generator iterator). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Before and after the colon in funciton defs.
by Ron Adam 24 Sep '11

24 Sep '11

(This idea may have been suggested before, because it seems too obvious to me.) How about if we remove the requirement that the colon be on the same line as the function name. And, what if the colon is then used to split the difference between the definition time, and call time code in function definitions. So that every thing before the colon is done at definition time. Everything after the colon is done at call time. def foo(...): """ doc string """ <function body> Then could become ... def foo(...) """ doc string """ # foo.__doc__ = """ doc string """ : <function body> I think this represents what is actually happening a bit better. One possibility for define time code is to have decorators listed that read in the order they are applied instead of bottom up. def foo(n) """ function to be decorated. """ @deco1 # foo = deco1(foo) The '@' notation still works. @deco2 # foo = deco2(foo) : <function body> Note, that putting the doc string after the decorators may be better as it would put the doc string on the decorated function instead of the original. I think there may be more things possible with this idea than the simple cases above. Cheers, Ron

24 82

static vs atdef, etc
by Jim Jewett 22 Sep '11

22 Sep '11

On Thu, Sep 22, 2011 at 10:34 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > * I'm sorry but the keyword atdef is ugly. The only proposal I like is > static, but I agree that it only makes sense to someone with a C-type > background. Even with a C background, static is a bad choice, because it has already been used for too many different meanings. (Admittedly, not all in C.) -jJ

2 1

win32 extensions
by Massimo Di Pierro 22 Sep '11

22 Sep '11

Just a question. What is the reason the win32 extensions are shipped separately? Is it a licensing issue or is there another reason? Massimo

11 40

Break the dominance of boolean values in boolean context
by Lukas Lueg 22 Sep '11

22 Sep '11

Hi, the proposal is to reduce the number of occurences where Python automatically and inevitably uses pure boolean objects (bool()), instead of relying on rich object behaviour. My goal is to give objects finer control over how they want to be interpreted in a boolean context. This reaches down as far as implementing the boolean operations as an object-protocol. In Python, no kind of object is in any way special. Instead of relying on what an object actually IS, we rely on what an object can DO. This approach of anti-discrimination is the source of great confidence about the language and it's aversion to black magic. The bool()-type however is a primus inter pares: In numerous locations, Python strongly favors this type of object either explicitly by casting to it or implicitly through the compiled code. Here are a three examples (protocol-, object- and code-based) where Python grinds a boolean context down to pure boolean values: In protocol behaviour: The documentation about how to implement __contains__ vaguely states that the function "should return true, false otherwise". In fact we may return any object that can be interpreted as a boolean, this is intented behaviour. However the implementation of COMPARE_OP in Python/ceval.c inevitably casts the result of "x in y" to Py_True or Py_False, throwing away our object and leaving us with this bool only. This kills any chance of producing useful objects in __contains__, even objects that may have a meaning outside the pure boolean context. For example: x = mycontainer(['foo', 'bar', 'bar', 'foo', 'foo']) y = 'foo' in mycontainer print y == True >> True print y >> mycontainer('foo', 'foo', 'foo') This has been discussed on python-ideas in mid-2010 (http://mail.python.org/pipermail/python-ideas/2010-July/007733.html) without distinct outcome. In object behaviour: Rich Comparision allows numeric- or set-like comparision in any way we like, returing any kind of object. The documentation clearly states that the returned object is only(!) interpreted as a boolean when used in a boolean context (__contains__ falls short here). For example: x = set((1,2,3,4,5)) y = set((2,3,4)) z = x > y print z == True >> True print z >> set([1,5]) # equivalent to x - y z = x < y print z == True >> False print z >> set([]) # equivalent to y - x print 3 > 2 >> 1 print (3 > 2) == True >> True In code behaviour: Why is it, that we can override the behaviour of arithmetic operations like "+, -, *, /", but not the boolean operations like "and, or, xor, not" ? Why do we force any value being generated in a boolean context like "if x == 1 or y == 1" to actually be of boolean type? The result of "(x/y) == 1" may be any kind of object coming from __eq__. However the "or"-operator here grinds them both down to being just True or False. This is due to the fact that the generated bytecode evaluates one expression at a time and does not keep track of the objects it got while doing so. The pure boolean behaviour arises from the use of JUMP_IF_FALSE/TRUE after each comparision. Instead of having boolean operations being a part of the language, we could implement them as a an extension to the Rich Comparision Protocol, giving rise to functions like object.__bor__, object.__bnot__ or object.__band__: The expression "if x == 1 or y == 1" would then become equivalent to tmp1 = x.__eq__(1) if tmp1: return tmp1 tmp2 = y.__eq__(1) if tmp2: return tmp2 return = tmp1.__bor__(tmp2) Likewise, the expression "if x == 1 and y == 1" would become tmp1 = x.__eq__(1) if not tmp1: return tmp1 tmp2 = y.__eq__(1) if not tmp2: return tmp2 return tmp1.__band__(tmp2) The object-example from above now tells us how boolean behaviour and arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" is True because "set([1,5]).__bor__(set([]))" is the same as "set([1,5]) + set([])" and equivalent to True in a boolean context. Likewise, "(setA > setB) and (setA < setB)" is False because "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA > setB) == (setA - setB) == (setA & setB)". We can't do this with boolean operations being part of the language only. Summing all up, I really think that we should break the dominance of bool() and take a look at how we can implement boolean contexts without relying on boolean values all the time. None of this can be implemented without breaking at least the CPython API. For example, the behaviour of __contains__ can't be changed in the proposed way without changing the signature of "int PySequence_Contains()" to "PyObject* PySequence_Contains()".

25 81

Re: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk
by Glyph Lefkowitz 20 Sep '11

20 Sep '11

On Sep 19, 2011, at 3:12 AM, Godson Gera wrote: > On Mon, Sep 19, 2011 at 12:03 AM, Glyph Lefkowitz <glyph(a)twistedmatrix.com> wrote: > > Actually, the Twisted developers also found this to be a terrible idea, and doesn't really do it any more. I definitely understand the appeal of such a system, but it has far too many problems to work well in Python, and even in Smalltalk (which was built from the start to work like this) it was the source of a number of long-term systematic problems with the ecosystem. > > I willfully leftout the detail that Twisted is not doing it any more :) . So that the person who asked this won't get discouraged, if he has time and energy to pursue it and make some thing new, I thought all the Python community may have another battery called persistance included. Yes, and I put it in in the hopes that they are discouraged :-). There's a staggering number of really good ideas out there for people to work on; why get stuck re-hashing the bad ones? Of course, maybe they've got a really fantastic idea about how a persistent python might work, but they'll have to face more obstacles than this one comment in order to get it done, so it shouldn't make that much of a difference. I think it's my responsibility, as someone who has tried this idea, to make sure that others attempting it know that it's problematic. As Clay Shirky put it: "... learning from experience is the worst possible way to learn something. Learning from experience is one up from remembering. That's not great. The best way to learn something is when someone else figures it out and tells you: 'Don't go in that swamp. There are alligators in there.'" So, trust me: there are alligators in that pickle. -glyph

4 3

Re: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk
by Glyph Lefkowitz 19 Sep '11

19 Sep '11

On Sep 18, 2011, at 4:55 AM, Godson Gera wrote: > Twisted has some feature like that implemented using pickles or some thing. It meant to save the state of the program during restart. I am not sure if that's what you are after. http://twistedmatrix.com > Actually, the Twisted developers also found this to be a terrible idea, and doesn't really do it any more. I definitely understand the appeal of such a system, but it has far too many problems to work well in Python, and even in Smalltalk (which was built from the start to work like this) it was the source of a number of long-term systematic problems with the ecosystem. -glyph

2 1

truncate sequences in pretty-print ?
by Steven Samuel Cole 18 Sep '11

18 Sep '11

hello, i use pprint quite a bit during development to give me quick insight into what is going on inside my application. however, when there's any sequences involved, the output becomes less useful the longer these sequences are - i often find myself scrolling around (or even searching) in the terminal window, trying to find the bit of output i am interested in. imo, it would be great if pprint had a parameter 'max_len' or so that reduces output of every sequence to a maximum and inserts something like '...' to indicate truncation, e.g. {'my key': ['my list item 01', 'my list item 02', 'my list item 03', 'my list item 04', 'my list item 05', 'my list item 06', 'my list item 07', 'my list item 08', '...', 'my list item 10',]} somewhat comparable to the '...' already printed when a structure is more deeply nested than you want to know right now. what do you think ? cheers, ssc

3 2

Re: [Python-ideas] allow line break at operators
by MRAB 18 Sep '11

18 Sep '11

On 04/09/2011 00:22, Yingjie Lan wrote: > On 04/09/2011 03:04, MRAB wrote: >> I think that the rules would be: >> >> If a line ends with a colon and the next line is indented, then >> it's the start of a block, and the following lines which belong to >> that block have the same indent. >> >> If a line doesn't end with a colon but the next line is indented, >> then it's the start of a continuation, and the following lines >> which belong to that continuation have the same indent. >> >> In both cases there could be blocks nested in blocks and possibly >> continuations nested in continuations, as well as blocks nested in >> continuations and continuations nested in blocks. >> >> I'm not sure what the effect would be if you had mis-indented >> lines. For example, if a line was accidentally indented after a >> comment, then it would be treated as part of the comment. It's in >> cases like those that syntax colouring would be helpful. It would >> be a good idea to use an editor which could indicate in some way >> when a line is a continuation. >> > Thanks, I think that's the rule described in its full glory. > Currently I am not quite sure of the use case for continuation > nested in continuation -- it seems to be still a single continuation, > but it allows for some additional freedom in formatting the continued > line. Do you have other use cases for that? > I don't have a use-case, I was just wondering whether in this: first second third fourth "third" is a continuation, giving this: first second third fourth which has 2 continuations, leading to this: first second third fourth > For the case of mis-indentation, as demonstrated in your scenario, I > think it is better that the rule is not applied to a comment > continued onto the next line. The effect of a '#' only carries to > the end of a line, if one would like the next line to be a comment, > just use another '#'. It remains to consider a mis-indentation that > only involves code lines. However, that is not a new problem, so we > should not worry (it is like a sunken cost). > As well as still limiting a comment to a line, I'd also still limit a string literal (except a triple-quoted string literal) to a line.

7 13

Re: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk
by Guido van Rossum 18 Sep '11

18 Sep '11

[BCC python-dev, +python-ideas] Funny you should mention this. ABC, Python's predecessor, worked like this. However, it didn't work out very well. So, I'd say you're about 30 years too late with your idea... :-( --Guido On Sat, Sep 17, 2011 at 8:05 AM, Albert Zeyer <albzey(a)googlemail.com> wrote: > Hi, > > I was thinking about a persistent Python interpreter system. I.e. you > start a Python interpreter instance and you load and create all your > objects, classes and code in there (or load it in there from other > files). > > The basic idea is that you wont restart your Python script, you would > always modify it on-the-fly. Or a bit less extreme: You would at least > have the possibility with this to do this (like just doing minor > changes). Also, if your PC halts for whatever reason, you can continue > your Python script after a restart. > > This goes along my other recent proposal to store the AST of > statements in the related code objects > (http://thread.gmane.org/gmane.comp.python.devel/126754). An internal > editor could then edit this AST and recompile the code object. > > For the persistance, there would be an image file containing all the > Python objects. > > All in all, much like most Smalltalk systems. > > --- > > Has anyone done something like this already? > > --- > > There are a few implementation details which are not trivial and there > doesn't seem to be straight forward solutions, e.g. most generally: > > * How to implement the persistance? > * How to handle image compatibility between CPython updates? Even possible? > > Regards, > Albert > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido)

5 4

return value of yield expressions
by H. Krishnan 16 Sep '11

16 Sep '11

Hi, I should start off by saying that my knowledge of PEP342 is less than a week old, and I cannot claim any use-case for what I am about to propose. The PEP itself says "In effect, a yield-expression is like an inverted function call; the argument to yield is in fact returned (yielded) from the currently executing function, and the "return value" of yield is the argument passed in via send()." One could, I guess, wrap a function to be 'called' via this mechanism: def genwrap(func): def gen(): ret = None while True: args, kwds = (yield ret) ret = func(*args, **kwds) del args, kwds g = gen() g.next() return g f = genwrap(func) f.send( (args, kwds) ) However, 'send' takes only one argument (and hence the poor syntax in the last statement). Has the option of the return value of the yield expression treated very much like function arguments been discussed? (a1, a2, a3 = default, *args, **kwds) = (yield ret) 'send' could support positional and keyword arguments. I guess this particular form would break backward compatibility but there might be other alternatives. Regards, Krishnan

9 32

Object interface to path names
by David Townshend 16 Sep '11

16 Sep '11

I've recently been developing a tool to track changes to a fairly large structured file system, and in the process got to thinking about how working with path names could be improved. The problems I've had with using just os and os.path have lead to three objectives of any new implementation: 1. Cleaner handling of paths names, specifically constructing path names without the need for a lot of nested os.path.join() and os.split() functions. 2. Allow a validation of paths names based on predefined rules. (Although this requirement might be very specific to my use case) 3. Allow caching of file attribute data so that queries do not have to wait the disk or network to respond (although at the cost of accuracy). The first can be met with behaviour as follows, basically handling paths as containers of sub-paths: >>> root = Path('/usr') >>> print([n for n in root]) ['bin', 'local', 'lib', 'share'] >>> print([n for n in root.dirs]) ['bin', 'local', 'lib', 'share'] >>> print([n for n in root.files]) [] >>> root['local/share'] Path('/usr/local/share') >>> root['/usr/local/share'] Path('/usr/local/share') >>> share = Path('/usr/local/share') >>> share in root True The second can be met by allowing Path to be subclassed and defining factory functions for the creation of sub-paths: >>> root = Path('/usr') >>> root.factory = my_factory_function >>> root['local/share'] MyPath('/usr/local/share') The third can be met be allowing all disk calls to be asynchonous: >>> path = Path('/home/david/newfile', async=True) >>> path.touch() >>> path.exists() False >>> time.sleep(2) >>> path.exists() True This could all be implemented in pure python using os and os.path, and threading for asynchonous calls. I haven't yet thought through a complete specification for Path, but I image it would need to contain functions such as exists(), isfile(), isdir(), stat(), walk(), and allow iterator access to children. Does anyone else see a usefulness for this? Regards David

6 10

threading : AttributeError: 'NoneType' object has no attribute 'start'
by Sagar Neve 14 Sep '11

14 Sep '11

I am new to the python and I need a small script to be written using threads which reads the specific files present in a directory and does some selections of lines and then write those lines into different files. However the error I am getting is: AttributeError: 'NoneType' object has no attribute 'start' Here is the code. Can somebody please help me. This is very urgent. def getFiles(fname): fname=preprocessLogFileDir + "/" + fname print fname fin = open(fname, "r") line=fin.readline() print line .... # some extraction logic. fout = open("out-single-file/1xx_2xx_total", "w") fout.write(line) fout.close() files=os.listdir(preprocessLogFileDir) cnt=1; for fname in files: print fname if fnmatch.fnmatch(fname,'*.log'): #and os.path.isfile(fname): thr = "thr" + str(cnt) print thr thr=getFiles(fname) thr.start() thr.join cnt = cnt + 1 else: print "\npreprocessLogFileDir/fname is either a directory or does not end with 'log'." print "Probabally not a pre-process file. Ignoring the file.\n\n"

4 4

List Revolution
by Christopher King 13 Sep '11

13 Sep '11

The first element in a list is element zero, the second is one, the third it two, and so on. This some times confuses newbies to the language or programming in general. This system was invited when single bits where precious. It's time to update. Keep in mind this is something for version 4, since its not reverse compatible. I say we make the first element 1, second 2, third 3, and so on. Other languages would follow. We are python, made for easiness and readability, and we are in the age where you can't even get something as small as a kilobyte USB. We must make first one, second 2, and third 3, like it is supposed to be. I give this: *+1*

24 38

Re: [Python-ideas] List Revolution
by Matt Billenstein 12 Sep '11

12 Sep '11

Ugh, I hate the idea of library code having to think about dealing with the possibility that you chose 42 as the starting index for some list you passed into a function in said library. Then I suppose we'll have to extend this to strings and tuples and so forth as well? Seriously, any further "solutions" for this non-problem are just mental masturbation -- please stop. m On Sun, Sep 11, 2011 at 11:05:28PM -0700, mike.w.meyer(a)gmail.com wrote: > Ok, how about enhancing the list type to allow specifying an offset, > defaulting to 0? The offset should be positive, and that many elements simple > aren't there. Asking for an element that isn't there, or a slice including > such an element, raises value error exception. The iterators & etc for this > list would all start at offset instead of zero. > -- > Sent from my Android tablet with K-9 Mail. Please excuse my brevity. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Matt Billenstein matt(a)vazor.com http://www.vazor.com/

1 0

Re: [Python-ideas] List Revolution
by Massimo Di Pierro 12 Sep '11

12 Sep '11

Here is my proposal which hopefully would settle the issue to rest class List(list): """ >>> a=List() >>> for i in range(100000): a.append(i) >>> print a.first 0 >>> print a.second 1 >>> print a.third 2 >>> print a.twentieth 19 >>> print a.twentysecond 21 >>> print a.onehundredthirtyfifth 134 >>> print a.onethousandfivehundredandthirtyeighth 1537 """ def __getattr__(self,name): import re if name.endswith('first'): name = name[:-5]+'one' elif name.endswith('second'): name = name[:-6]+'two' elif name.endswith('third'): name = name[:-5]+'three' elif name.endswith('fth'): name = name[:-3]+'ve' elif name.endswith('hth'): name = name[:-3]+'th' elif name.endswith('ieth'): name = name[:-4]+'y' elif name.endswith('th'): name = name[:-2] subs = { 'one':'+1', 'two':'+2', 'three':'+3', 'four':'+4', 'five':'+5', 'six':'+6', 'seven':'+7', 'eigth':'+8', 'nine':'+9', 'ten':'+10', 'eleven':'+11', 'twelve':'+12', 'thirteen':'+13', 'fourteen':'+14', 'fiftheen':'+15', 'sixteen':'+16', 'seventeen':'+17', 'eighteen':'+18', 'nineteen':'+19', 'ten':'+10', 'twenty':'+20', 'thirty':'+30', 'fourty':'+40', 'fifthy':'+50', 'sixty':'+60', 'seventy':'+70', 'eighty':'+80', 'ninety':'+90', 'hundred':')*100+(', 'thousand':')*1000+(', 'million':')*1000000+(', 'billion':')*1000000000+(', 'trillion':')*100000000000+(', 'and',''} for key,value in subs.items(): name = name.replace(key,value) if '(' in name: name='('+name+')' name.replace('()','1') if not re.compile('[\d\+\*]+').match(name): return AttributeError try: return self[eval(name)-1] except: raise AttributeError I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. Massimo

6 11

Columns for pprint.PrettyPrinter
by Julian Berman 11 Sep '11

11 Sep '11

For the most part I find pprint to be lovely. Simple and does a useful thing that I don't want to have to write every time, especially if I'm using it just to debug. However, I often find that I'm using it to print out a long-length singly nested structure whose elements' reprs are not really that long, for which its output is cumbersome. Try to pprint.pprint(range(100)). Instead of seeing a long really skinny column, it'd be nice if it could quickly provide a way (combined with the width arg that it already takes) to split up the elements of the structure within the width, so that instead of seeing things like >>> pprint.pprint(range(30)) [0, 1, 2, ... ] it could be coerced into something like >>> pprint.pprint(range(30), columns=5) [0, 1, 2, 3, 4 3, 4, 5, 6, 7, ... ] or for something nested, which I'm less thrilled with, and haven't thought out how to implement unless you have a somewhat balanced structure, but for posterity: {"foo" : {"bar" : 1, {"hello" : 2, {"other" : 1, "baz" : 2, "world" : 1}, "thing" : 2, "foo" : 3}, "here" : 3}, ... } Obviously it's meant to be simple, the comment at the top of the module even says so, and doing something like ^ is easy enough, but for what it's good for (saving me from having to write code that makes my objects easier to debug by displaying them nicely), just making it do a bit more would make life easier.

3 2

relaxing keyword usage restrictions
by H Krishnan 11 Sep '11

11 Sep '11

Hi, Apologies if this has been asked already: why can't the usage of at least some keywords as identifiers be allowed with the context of the usage being used to identify if a keyword or a variable is being referred to. For example, if = 3 # if is identifier if if = 3: # first if is keyword, second if is identifer A more complex usage may require some rules: a = not or and and ==> parse left to right to interpret as a = ( (not or) and (and) ). 'or' and the last 'and' are identifiers While some of the usage may be confusing to the user, that is not for the language to dictate but for the user to choose. Almost all keywords have specific semantics which (in my limited understanding) do not conflict with their usage as identifiers. Regards, Krishnan

23 51

Re: [Python-ideas] List Revolution
by Guido van Rossum 10 Sep '11

10 Sep '11

On Sat, Sep 10, 2011 at 12:07 PM, Massimo Di Pierro <mdipierro(a)cs.depaul.edu> wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the Python parser instead of reinventing the wheel. It is less error prone and easier to understand. It is a python keyword. It is unsafe (as the bug in your first version inadvertently illustrated :-). I don't actually quite follow what your example does (sorry), but, even though it is less powerful than exec(), eval() invokes the full Python parser+compiler, which can take quite a bit of time to start up and uses a fair bit of memory. Also in other Python versions this is often much slower than in CPython, or has a big startup overhead. But, of course, it is great for quick demos! -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Python-ideas] List Revolution
by Massimo Di Pierro 10 Sep '11

10 Sep '11

On Sep 10, 2011, at 2:22 PM, Matt Billenstein wrote: > On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: >> Not to open a can of worms but why not use eval in this example? >> It is faster. It is 5x more compact. It allows me to take advantage of the >> Python parser instead of reinventing the wheel. It is less error prone and >> easier to understand. It is a python keyword. > > I wouldn't worry too much about the implementation details of a clearly insane > idea... agreed > And eval generally has high "code-smell" -- it can lead to code injection exploits. Not in this example. So why not? Any code can introduce vulnerabilities if incorrect.

2 2

Re: [Python-ideas] List Revolution
by Matt Billenstein 10 Sep '11

10 Sep '11

On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the > Python parser instead of reinventing the wheel. It is less error prone and > easier to understand. It is a python keyword. I wouldn't worry too much about the implementation details of a clearly insane idea... And eval generally has high "code-smell" -- it can lead to code injection exploits. m -- Matt Billenstein matt(a)vazor.com http://www.vazor.com/

1 0

Re: [Python-ideas] List Revolution
by Massimo Di Pierro 10 Sep '11

10 Sep '11

On Sep 10, 2011, at 2:07 PM, Massimo Di Pierro wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the Python parser instead of reinventing the wheel. It is less error prone and easier to understand. It is a python keyword. > > # with eval > if not re.compile('^[\*\+\d]+$').match(name): raise AttributeError > try: number = eval(name) > except: raise AttributeError > > # without eval > tokenizer = re.compile('\+|$\+|\(|$|\*|[\d]+') > stack = [0] > while name: > token = tokenizer.match(name) > if not token: raise AttributeError > elif token.group() in ('(','(+'): stack.append(0) > elif token.group().isdigit(): stack.append(int(token.group())) > elif token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] > elif token.group()=='*': > name = name[1:] > token = tokenizer.match(name) > if not token or not token.group().isdigit(): raise AttrbuteError > stack[-1:]=[stack[-1]*int(token.group())] > name = name[token.end():] > number = sum(stack) > > > > Massimo > > > On Sep 10, 2011, at 1:50 PM, Massimo Di Pierro wrote: > >> Of course. It was a proof of concept. Here is with an inline parser: >> >> class List(list): >> """ >>>>> a=List() >>>>> for i in range(100000): a.append(i) >>>>> print a.first >> 0 >>>>> print a.second >> 1 >>>>> print a.third >> 2 >>>>> print a.twentieth >> 19 >>>>> print a.twentysecond >> 21 >>>>> print a.onehundredthirtyfifth >> 134 >>>>> print a.onethousandfivehundredthirtyeighth >> 1537 >> """ >> def __getattr__(self,name): >> if name.endswith('first'): name = name[:-5]+'one' >> elif name.endswith('second'): name = name[:-6]+'two' >> elif name.endswith('third'): name = name[:-5]+'three' >> elif name.endswith('fth'): name = name[:-3]+'ve' >> elif name.endswith('hth'): name = name[:-3]+'th' >> elif name.endswith('ieth'): name = name[:-4]+'y' >> elif name.endswith('th'): name = name[:-2] >> subs = [ >> ("eleven","+11"), >> ("twelve","+12"), >> ("thirteen","+13"), >> ("fourteen","+14"), >> ("fiftheen","+15"), >> ("sixteen","+16"), >> ("seventeen","+17"), >> ("eighteen","+18"), >> ("nineteen","+19"), >> ("ten","+10"), >> ("twenty","+20"), >> ("thirty","+30"), >> ("fourty","+40"), >> ("fifty","+50"), >> ("sixty","+60"), >> ("seventy","+70"), >> ("eighty","+80"), >> ("ninety","+90"), >> ("one","+1"), >> ("two","+2"), >> ("three","+3"), >> ("four","+4"), >> ("five","+5"), >> ("six","+6"), >> ("seven","+7"), >> ("eigth","+8"), >> ("nine","+9"), >> ("ten","+10"), >> ("hundred",")*100+("), >> ("thousand",")*1000+("), >> ("million",")*1000000+("), >> ("billion",")*1000000000+("), >> ("trillion",")*100000000000+("), >> ("and","")] >> for key,value in subs: name = name.replace(key,value) >> if '(' in name: name='('+name+')' >> name.replace('()','1') >> ## parser >> import re >> tokenizer = re.compile('\+|$\+|\(|$|\*|[\d]+') >> stack = [0] >> while name: >> token = tokenizer.match(name) >> if not token: raise AttributeError >> if token.group() in ('(','(+'): stack.append(0) >> if token.group().isdigit(): stack.append(int(token.group())) >> if token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] >> if token.group()=='*': >> name = name[1:] >> token = tokenizer.match(name) >> if not token or not token.group().isdigit(): raise AttrbuteError >> stack[-1:]=[stack[-1]*int(token.group())] >> name = name[token.end():] >> number = sum(stack) >> ## end parser >> try: return self[number-1] >> except: raise IndexError >> >> >> >> >> On Sep 10, 2011, at 1:00 PM, Guido van Rossum wrote: >> >>> I thought we were all just having a little bit of fun with what's >>> obviously the least-likely-to-be-accepted proposal to have hit >>> python-ideas in a long time; and now I see serious questions about it >>> on Twitter and Quora... Time to start adding smileys to all posts! :-) >>> >>> (Though seriously, Massimo, you should be able to implement the same >>> idea without using eval() at all.) >>> >>> --Guido >>> >>> On Sat, Sep 10, 2011 at 10:50 AM, Massimo Di Pierro >>> <massimo.dipierro(a)gmail.com> wrote: >>>> Anyway ... mine was not a serious proposal. I like python lists as they are. >>>> >>>> On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: >>>> >>>>>> Damn dollar signs. >>>>>> if not re.compile('^[\d\+\*]+$').match(name): return AttributeError >>>>> >>>>> '(2**1024)**(2**1024)' etc. >>>>> >>>>> Devin >>>>> >>>>> On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro >>>>> <massimo.dipierro(a)gmail.com> wrote: >>>>>> LOL. >>>>>> >>>>>> Damn dollar signs. >>>>>> if not re.compile('^[\d\+\*]+$').match(name): return AttributeError >>>>>> >>>>>> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >>>>>> >>>>>>> Massimo Di Pierro, 10.09.2011 18:35: >>>>>>>> [...] >>>>>>>> if not re.compile('[\d\+\*]+').match(name): return AttributeError >>>>>>>> [...] >>>>>>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>>>>>> >>>>>>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Python-ideas mailing list >>>>>>> Python-ideas(a)python.org >>>>>>> http://mail.python.org/mailman/listinfo/python-ideas >>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas(a)python.org >>>>>> http://mail.python.org/mailman/listinfo/python-ideas >>>>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas(a)python.org >>>> http://mail.python.org/mailman/listinfo/python-ideas >>>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >> >

1 0

Deprecating bytes.swapcase and friends [was: Maintenance burden of str.swapcase]
by Stephen J. Turnbull 07 Sep '11

07 Sep '11

This is all speculation and no hint of implementation at this point ... redirecting this subthread to Python-Ideas. Reply-To set accordingly. Nick Coghlan writes: > Heh, I knew as soon as I sent that message that someone would be able > to point out a counter example. I agree that RFC 822 (and > case-insensitive ASCII comparison in general) is enough to save > lower() and upper() and co, but what about this even further reduced > list of text-specific methods: > > 'capitalize' > 'istitle' > 'swapcase' > 'title' > > While case-insensitive comparison makes sense for wire level data, > where do these methods fit in, even when embedded ASCII text fragments > are involved? Well, 'capitalize' could theoretically be used to "beautify" RFC 822 field names, but realistically, to me they're a litmus test for packages I probably don't want on my system.<0.5 wink> I don't know if it's worth the effort to deprecate them, though. There is a school of thought (represented on python-dev by Philip Eby and Antoine Pitrou, among others, I would say) that says that text with an implicit encoding is still text if you can figure out what the encoding is, and the syntactically important tokens are invariably ASCII, which often is enough information to do the work. So if you can do some operation without first converting to str, let's save the cycles and the bytes (especially in bit-shoveling applications like WSGI)! I disagree, but "consenting adults" and all that. It occurs to me that the bit-shoveling applications would generally be sufficiently well-served with a special "codec" that just stuffs the data pointer in a bytes object into the latin1 member of the data pointer union in a PEP 393 Unicode object, and marks the Unicode object as "ascii-compatible", ie, anything ASCII can be manipulated as text, but anything non-ASCII is like a private character that Python doesn't know anything about, and can't do anything useful with, except delete or pass through verbatim (perhaps as a slice). This may be nonsense; I don't know enough about Python internals to be sure. And it would be a change to PEP 393, since the encoding of the 8-bit representation would no longer be Unicode. I wouldn't blame Martin one bit if he hated the idea in principle! On the other hand, the "Latin-1 can be used to decode any binary content" end-around makes that point moot IMO. This would give a somewhat safer way of doing that. But if feasible and a Pythonic implementation could be devised, that would take much of the wind out of the sails of the "implicitly it's ASCII text" crowd. The whole "it's inefficient in time and space to work with 'str'" argument goes away, leaving them with "it's verbose" as the only reason for not doing the conversion. I don't know if there would be any use case left for bytes at that point ... but that's clearly a py4k discussion.

3 2

xmlrpc.client to deprecate Binary for bytes?
by Masklinn 06 Sep '11

06 Sep '11

Not sure if this is an actual idea/feature request or a bug. Might be half/half In Python 2, many actual strings were "packaged" through the str type, so xmlrpclib had to provide an other wrapper (xmlrpclib.Binary) to differentiate between actual strings (to insert as-is) and binary blobs, to b64encode. Among Python 3 fixes are the stricter separation between bytestrings (bytes) and text strings (str), but xmlrpc.client *still* requires the usage of a Binary wrapper, and trying to encode a bytes instance just blows up: >>> xmlrpc.client.dumps((b'foo',)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 956, in dumps data = m.dumps(params) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 506, in dumps dump(v, write) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 530, in __dump f(self, value, write) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 569, in dump_string write(escape(value)) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 156, in escape s = s.replace("&", "&") TypeError: expected an object with the buffer interface >>> xmlrpc.client.dumps((xmlrpc.client.Binary(b'foo'),)) '<params>\n<param>\n<value><base64>\nZm9v\n\n</base64></value>\n</param>\n</params>\n' I think it would be worthwhile to do the following: * Have xmlrpc.client treat dumping `bytes` (and related objects, such as bytearray or memoryview) as if they were `Binary`-wrapped (by b64encoding them and shipping them as binary data, which they are) * Add a flag to xmlrpc.client's `ServerProxy` and `loads` to have binary data (<base64> nodes of xmlrpc) decode to `bytes` rather than `Binary` instances. This option would likely be `False` by default, at least to start with, so as not to break compatibility with Python 3.1 and 3.2 code. PS: as seen above, xmlrpc claims to require objects "with the buffer interface", Python's documentation (http://docs.python.org/py3k/library/stdtypes.html?highlight=memoryview#memo…) claims bytes and bytearray implement "the buffer protocol". Who is lying?

1 0

Improving this? - foo() takes at least 6 arguments (6 given)
by Mark Dickinson 05 Sep '11

05 Sep '11

On Sat, Sep 3, 2011 at 9:59 AM, Eli Stevens (Gmail) <wickedgrey(a)gmail.com> wrote: > On Thu, Sep 1, 2011 at 10:19 AM, Ethan Furman <ethan(a)stoneleaf.us> wrote: >>> Colleague: "So my code is apparently asking for 6 arguments and it >>> appears that I gave it 6 arguments" >>> Me: "Oh that - Yeah... your short one. It really wants 7.." >> >> How did you get this? > > Having seen this bite someone recently, I have an example readily at hand: > >>>> def f(a,b,c,d=None): pass > ... >>>> f(1,2,d=4) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: f() takes at least 3 arguments (3 given) Isn't this already fixed for Python 3.3? Python 3.3.0a0 (default:a8748022504f, Sep 2 2011, 12:32:36) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> def f(a, b, c, d=None): pass ... [63593 refs] >>> f(1, 2, d=4) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: f() missing 1 required positional argument: 'c' [63624 refs]

3 2

Re: [Python-ideas] allow line break at operators
by Matt Joiner 04 Sep '11

04 Sep '11

I guess the issue here is that you can't tell if an expression is complete without checking the indent of the following line. This is likely not desirable. On Thu, Sep 1, 2011 at 11:43 PM, Yingjie Lan <lanyjie(a)yahoo.com> wrote: > Hi Matt, > ======================================================= > From: Matt Joiner <anacrolix(a)gmail.com> > > The "trailing \" workaround is nonobvious. Wrapping in () is noisy and > already heavily used by other syntactical structures. > ======================================================= > How about only require indentation > to freely break lines? Here is an example: > x = firstpart * secondpart #line breaks here > + anotherpart #continue by indentation > + stillanother #continue on. > #until here, another line starts by dedentation > y = some_expression - another_one > All this would be completely compatible with former code, while > having almost free line breaking! Plus, indentation makes it pretty. > Really hope Python can have freedom in breaking lines. > Yingjie

5 5

Improving this? - foo() takes at least 6 arguments (6 given)
by Steven Klass 03 Sep '11

03 Sep '11

Hi all, Most of us who have been coding for awhile have often seen this error and understand the "self" concept. I was helping a junior colleague out and he came to me with this problem. Colleague: "So my code is apparently asking for 6 arguments and it appears that I gave it 6 arguments" Me: "Oh that - Yeah... your short one. It really wants 7.." That got me thinking and here is the post. Now while this seems simple to fix I know this isn't that simple and a healthy dose of introspection will be required. --- Steven Klass (480) 225-1112 sklass(a)pointcircle.com

4 3

Re: [Python-ideas] allow line break at operators
by Stephen J. Turnbull 03 Sep '11

03 Sep '11

Yingjie Lan writes: > Python uses indentation for blocks, and by the same mechanism, line > breaking can be accommodated without requiring parenthesis or > ending backslashes. Possibly, but now you have a problem that a dedent has ambiguous meaning. It might mean that you're ending a suite, or it might mean you're ending a continued expression. This probably can be disambiguated, but I don't know how easy that will be to do perfectly, including in reporting ill-formed programs. > Most people seems to like an indentation on the continuing lines, Most of the time, yes, but sometimes not. For example, in generating text, it's often useful to dedent substantially so you can have a nearly normal length line in the literal strings being concatenated. Or you might have a pattern like this: x = long_named_variable_a - long_named_variable_a_base + long_named_variable_b - long_named_variable_b_base which your parser would raise an error on, I presume. That's not freedom!<wink />

1 0

Re: [Python-ideas] allow line break at operators
by Stephen J. Turnbull 03 Sep '11

03 Sep '11

Yingjie Lan writes: > Have you considered line continuation by indentation? It seems to > meet the design principle. I think it is the most natural way to > allow free line breaking in Python. Briefly, yes, and I think it would need a lot of tuning and probably complex rules. Unlike statements, where everybody (except the judges of the Obfuscated C Contest) agrees on a simple rule: "In a control structure, the controlled suite should be uniformly indented one level", line breaking and indentation of long expressions is an art, and people have different opinions on "readability" and "beauty." Achieving a compromise that is workable even for a few major styles is likely to be annoying and bug-prone. Pretty much every program I write seems to have a continued list of data or a multi-line dictionary display as data. It's not unusual for me to comment the formal arguments in a function definition, or the parent classes of a class definition. The exception for parenthesized objects is something I depend on for what I consider good style. Of course I could use explicit continuation, but in a long table that's ugly and error-prone. Long expressions that need to be broken across lines, on the other hand, often indication that I haven't thought carefully enough about that component of the program, and an extra pair of parentheses or a terminal backslash just isn't that "heavy" or ugly in the context of such long expressions. For me, they're also pretty rare; many programs I write have no explicit continuations in them at all. YMMV, of course, but I find the compromise that Python arrived at to be very useful, and I must suppose that it was substantially easier to implement than "fully free" line breaking (whatever that means to you).

1 0

Re: [Python-ideas] aliasing
by Peio Borthelle 02 Sep '11

02 Sep '11

Le 31 août 2011 à 17:20, python-ideas-request(a)python.org a écrit : > I have often thought that would be a nice to have feature, but to be > honest, I have never found a use for it that was absolutely necessary. I > have always found another way to solve the problem. > > To make it work using just ordinary assignment syntax, as you suggest, > requires more than just an "alias" function. It would need changes to > the Python internals. Possibly very large changes. Without a convincing > use-case, I don't think that will happen. > > So even though I think this would be a neat feature to have, and > possibly even useful, I don't think it is useful enough to justify the > work needed to make it happen. > > -- > Steven I don't agree with you when you say there is no necessary use case, e.g. I have two images in a gui and they must stay at the same position (relatively one to the other) but follow the movements of the arrow keys (truthful!). You can't do it by using an intermediate list because the list object is mutable but his data not: ------------------------ >>> list = [3,] >>> a = list[0] >>> list[0] = 6 >>> a 3 ------------------------- I really don't see any other solution, even I'm sure it would be a twisted operation and I don't find it very pythonic... Amicalement, Peio

8 11

Re: [Python-ideas] IP addressing library, aka reviving PEP 3144 (Re: Add from __experimental__ import bla)
by Nick Coghlan 02 Sep '11

02 Sep '11

On Fri, Sep 2, 2011 at 8:03 AM, Peter Moody <pmoody(a)google.com> wrote: > I'll make an implementation with these changes this weekend. Great to hear! > It sounds like it's going to be clunky, but that's definitely with a grain or two of bias. Hopefully it won't be too inconvenient - in theory, it just means that when you derive the network details from a host address and masking information you can't throw the original host address information away any more (since IPNetwork own't be keeping it around). > Documentation isn't my strong suit, but assuming the implementation is > palatable to all involved, that can be forthcoming as well. Splitting the API up a little should actually make the documentation task itself easier. For folks that are already familiar with networking concepts, it isn't that hard to mentally map the "IPNetwork" name as meaning "either an actual IP network or else an arbitrary host/netmask pair". We might grumble about it (*cough*), but we wouldn't be confused once we made the connection. However, once ipaddr goes into the standard library, its module documentation is going to be the first introduction many future Python programmers will have to the details of how IP addressing actually works. We'll refer them off to other resources, of course, but having clean semantics in the API itself is important in helping people that are *not* already networking experts use the module correctly. That means having the class model map cleanly to the real world entities it represents, and while IP networks and specific interfaces on those networks are both represented accurately by the combination of an IP address with masking information, they're *not* the same kind of thing. It's the difference between using Pair(a, b) for both 2-D coordinates and rational numbers vs using separate Point(x, y) and Rational(n, d) classes. Yes, the information stored in both cases is the same, but the semantics assigned to that information changes. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-ideas] Create a StringBuilder class and use it everywhere
by k_bx 02 Sep '11

02 Sep '11

> This doesn't seem nicer to read and write to me than the list form. I > also do not see any reason to believe it will stop people from doing > it the quadratic way if the ubiquitous make-a-list-then-join idiom > does not. The whole point is that people don't write u''.join idiom just because they don't know it is slow. And when they see StringBuilder -- they can ask themselves "why is he using that". I don't mind using u''.join, but it just doesn't make people think about speed at all. The most popular (as from what I can see) thing right now where people start seeing that += is slow is when they try to do that on PyPy (which doesn't have hack like CPython, who is still slow) and ask "why my pypy code is sooooo slow". With StringBuilder used widely that would not be the case.

14 19

Add from __experimental__ import bla [was: Should we move to replace re with regex?]
by Dj Gilcrease 01 Sep '11

01 Sep '11

In the thread about replacing re with regex someone mentioned adding to __future__ which isnt a great idea as future APIs are already solidified, they just live there to give developer time to adapt their code. The idea of a __experimental__ area is good for any pep's or stliib additions that are somewhat controversial (API isnt agreed on, code may take a while to integrate properly, developer wants some time to hash out any edge case bugs or API clarifications that may come up in large scale testing, etc). __experimental__ should emit a warning on import that says anything in here may change or be removed at any time and should not be used in stable code. __experimental__ features should behave the same as __future__ in that they can add new keywords or semantics to the existing language __experimental__ features can move directly to the stlib or builtins if they do not add new keywords and/or are backwards compatible with the feature they are replacing. Otherwise they move into __future__ for how ever many releases are deemed reasonable time for developers to adapt their code.

20 52

Re: [Python-ideas] aliasing
by Christopher King 01 Sep '11

01 Sep '11

Wrong list again On Thu, Sep 1, 2011 at 4:10 PM, Christopher King <g.nius.ck(a)gmail.com>wrote: > ------------------------ >> >>> list = [3,] >> >>> a = list >> >>> list[0] = 6 >> >>> a[0] >> 3 >> ------------------------- >> > Slight error in my code. It should be. > ------------------------ > >>> list = [3,] > >>> a = list > >>> list[0] = 6 > >>> a[0] > 6 > ------------------------- >

1 0