Mailman 3 March 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

Make __reduce__ to correspond to __getnewargs_ex__
by Neil Girdhar June 8, 2014

June 8, 2014

Currently __reduce__<http://docs.python.org/3.4/library/pickle.html#object.__reduce__>returns up to five things: (1) self.__new__ (or a substitute) (2) the result of __getnewargs__<http://docs.python.org/3.4/library/pickle.html#object.__getnewargs__>, which returns a tuple of positional arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> , (3) the result of __getstate__<http://docs.python.org/3.4/library/pickle.html#object.__getstate__>, which returns an object to be passed to __setstate__<http://docs.python.org/3.4/library/pickle.html#object.__setstate__> (4) an iterator of values for appending to a sequence (5) an iterator of key-value pairs for setting on a string. Python 3.4 added the very useful (for me) __getnewargs_ex__<http://docs.python.org/3.4/library/pickle.html#object.__getnewargs_ex__>, which returns a pair: (1) a tuple of positional arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> (2) a dict of keyword arguments for __new__<http://docs.python.org/3.4/reference/datamodel.html#object.__new__> Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__. Best, Neil

3 7

OrderedDict literals
by Anthony Towns May 9, 2014

May 9, 2014

Hi, I was re-reading some old threads about ordereddict literals to see if any of them had gotten anywhere. Amongst them, I came across a post by Tim Delaney: https://mail.python.org/pipermail/python-ideas/2011-January/009049.html that mentioned an odict literal of ['key': 'value', 'key2': 'value2'] could be confused with slice notation. >From a syntax point-of-view, that doesn't seem to be true (as mentioned in some of the replies to that thread), but it seems like you can abuse the similarity to make it a little easier to declare ordereddicts: from collections import OrderedDict class ODSlicer(object): def __getitem__(self, key): if type(key) is slice: key = [key] od = OrderedDict() for k in key: if type(k) is slice: od[k.start] = k.stop else: od[k] = k return od od = ODSlicer() print(od[1:2]) print(od["a":"b", "c":5]) print(od['a':'b', 'c':'d', ..., 'a':10, 'e':'f']) You could then replace: mydict = { 'foo': 'bar', 'baz': 'quux', } with: mydict = od[ 'foo': 'bar', 'baz': 'quux', ] if you need to convert a hardcoded dict into a hardcoded ordereddict. Works fine in python2.7 and python3.4. At this point, I'd like to note in my defence that this isn't called the python-good-ideas list :) What's the actual objection to supporting ['foo': 'bar'] odict literals? I saw Guido gave a -100, way back in the day, but no actual explanation for why it was distasteful? https://mail.python.org/pipermail/python-ideas/2009-June/004924.html Cheers, aj -- Anthony Towns <aj(a)erisian.com.au>

6 7

Fwd: [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power
by Nathaniel Smith April 9, 2014

April 9, 2014

Hi all, In the process of cleaning out old design... peculiarities... in numpy, I happened to look into the history of attempts to add syntax for matrix multiplication to Python, since the lack of this is (as you'll see) at the root of various intractable problems we have. I was pretty surprised; it turns out that even though numerical folks have been whinging about missing this operator for ~15 years, the only two attempts that have been made to add it were: PEP 211, which instead adds an operator for itertools.product, aka, "maybe we can sneak matrix multiply past Guido in some sort of... large, wooden rabbit..." and PEP 225, aka "let's add 12 new operators and figure out what to do with them later" I'd have rejected these too! So I thought, maybe we should try the radical tactic of writing down what we actually want, carefully explaining why we want it, and then asking for it. And at least this way, if it gets rejected, we'll know that it was rejected for the right reasons... You'll notice that this draft is rather more developed than the average first-round PEP posting, because it's already been the rounds of all the various numerical package mailing lists to build consensus; no point in asking for the wrong thing. Don't let that slow you down, though. I think what we have here is fairly convincing and covers a lot of the design space (at least it convinced me, which I wasn't sure of at the start), but I'm still totally open to changing anything here based on comments and feedback. AFAICT the numerical community would walk over hot coals if there were an infix matrix multiplication operator on the other side. (BTW, since this is python-ideas -- have you considered adding hot coals to python 3? It might do wonders for uptake.) ...Anyway, the point is, I'm sure I can wrangle them into accepting any useful suggestions or other changes deemed necessary by the broader Python community. -n --- [begin draft PEP -- monospace font recommended] --- PEP: XXXX Title: Dedicated infix operators for matrix multiplication and matrix power Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com> Status: Draft Type: Standards Track Python-Version: 3.5 Content-Type: text/x-rst Created: 20-Feb-2014 Post-History: Abstract ======== This PEP proposes two new binary operators dedicated to matrix multiplication and matrix power, spelled ``@`` and ``@@`` respectively. (Mnemonic: ``@`` is ``*`` for mATrices.) Specification ============= Two new binary operators are added to the Python language, together with corresponding in-place versions: ======= ========================= =============================== Op Precedence/associativity Methods ======= ========================= =============================== ``@`` Same as ``*`` ``__matmul__``, ``__rmatmul__`` ``@@`` Same as ``**`` ``__matpow__``, ``__rmatpow__`` ``@=`` n/a ``__imatmul__`` ``@@=`` n/a ``__imatpow__`` ======= ========================= =============================== No implementations of these methods are added to the builtin or standard library types. However, a number of projects have reached consensus on the recommended semantics for these operations; see `Intended usage details`_ below. Motivation ========== Executive summary ----------------- In numerical code, there are two important operations which compete for use of Python's ``*`` operator: elementwise multiplication, and matrix multiplication. In the nearly twenty years since the Numeric library was first proposed, there have been many attempts to resolve this tension [#hugunin]_; none have been really satisfactory. Currently, most numerical Python code uses ``*`` for elementwise multiplication, and function/method syntax for matrix multiplication; however, this leads to ugly and unreadable code in common circumstances. The problem is bad enough that significant amounts of code continue to use the opposite convention (which has the virtue of producing ugly and unreadable code in *different* circumstances), and this API fragmentation across codebases then creates yet more problems. There does not seem to be any *good* solution to the problem of designing a numerical API within current Python syntax -- only a landscape of options that are bad in different ways. The minimal change to Python syntax which is sufficient to resolve these problems is the addition of a single new infix operator for matrix multiplication. Matrix multiplication has a singular combination of features which distinguish it from other binary operations, which together provide a uniquely compelling case for the addition of a dedicated infix operator: * Just as for the existing numerical operators, there exists a vast body of prior art supporting the use of infix notation for matrix multiplication across all fields of mathematics, science, and engineering; ``@`` harmoniously fills a hole in Python's existing operator system. * ``@`` greatly clarifies real-world code. * ``@`` provides a smoother onramp for less experienced users, who are particularly harmed by hard-to-read code and API fragmentation. * ``@`` benefits a substantial and growing portion of the Python user community. * ``@`` will be used frequently -- in fact, evidence suggests it may be used more frequently than ``//`` or the bitwise operators. * ``@`` allows the Python numerical community to reduce fragmentation, and finally standardize on a single consensus duck type for all numerical array objects. And, given the existence of ``@``, it makes more sense than not to have ``@@``, ``@=``, and ``@@=``, so they are added as well. Background: What's wrong with the status quo? --------------------------------------------- When we crunch numbers on a computer, we usually have lots and lots of numbers to deal with. Trying to deal with them one at a time is cumbersome and slow -- especially when using an interpreted language. Instead, we want the ability to write down simple operations that apply to large collections of numbers all at once. The *n-dimensional array* is the basic object that all popular numeric computing environments use to make this possible. Python has several libraries that provide such arrays, with numpy being at present the most prominent. When working with n-dimensional arrays, there are two different ways we might want to define multiplication. One is elementwise multiplication:: [[1, 2], [[11, 12], [[1 * 11, 2 * 12], [3, 4]] x [13, 14]] = [3 * 13, 4 * 14]] and the other is `matrix multiplication`_: .. _matrix multiplication: https://en.wikipedia.org/wiki/Matrix_multiplication :: [[1, 2], [[11, 12], [[1 * 11 + 2 * 13, 1 * 12 + 2 * 14], [3, 4]] x [13, 14]] = [3 * 11 + 4 * 13, 3 * 12 + 4 * 14]] Elementwise multiplication is useful because it lets us easily and quickly perform many multiplications on a large collection of values, without writing a slow and cumbersome ``for`` loop. And this works as part of a very general schema: when using the array objects provided by numpy or other numerical libraries, all Python operators work elementwise on arrays of all dimensionalities. The result is that one can write functions using straightforward code like ``a * b + c / d``, treating the variables as if they were simple values, but then immediately use this function to efficiently perform this calculation on large collections of values, while keeping them organized using whatever arbitrarily complex array layout works best for the problem at hand. Matrix multiplication is more of a special case. It's only defined on 2d arrays (also known as "matrices"), and multiplication is the only operation that has a meaningful "matrix" version -- "matrix addition" is the same as elementwise addition; there is no such thing as "matrix bitwise-or" or "matrix floordiv"; "matrix division" can be defined but is not very useful, etc. However, matrix multiplication is still used very heavily across all numerical application areas; mathematically, it's one of the most fundamental operations there is. Because Python syntax currently allows for only a single multiplication operator ``*``, libraries providing array-like objects must decide: either use ``*`` for elementwise multiplication, or use ``*`` for matrix multiplication. And, unfortunately, it turns out that when doing general-purpose number crunching, both operations are used frequently, and there are major advantages to using infix rather than function call syntax in both cases. Thus it is not at all clear which convention is optimal, or even acceptable; often it varies on a case-by-case basis. Nonetheless, network effects mean that it is very important that we pick *just one* convention. In numpy, for example, it is technically possible to switch between the conventions, because numpy provides two different types with different ``__mul__`` methods. For ``numpy.ndarray`` objects, ``*`` performs elementwise multiplication, and matrix multiplication must use a function call (``numpy.dot``). For ``numpy.matrix`` objects, ``*`` performs matrix multiplication, and elementwise multiplication requires function syntax. Writing code using ``numpy.ndarray`` works fine. Writing code using ``numpy.matrix`` also works fine. But trouble begins as soon as we try to integrate these two pieces of code together. Code that expects an ``ndarray`` and gets a ``matrix``, or vice-versa, may crash or return incorrect results. Keeping track of which functions expect which types as inputs, and return which types as outputs, and then converting back and forth all the time, is incredibly cumbersome and impossible to get right at any scale. Functions that defensively try to handle both types as input and DTRT, find themselves floundering into a swamp of ``isinstance`` and ``if`` statements. PEP 238 split ``/`` into two operators: ``/`` and ``//``. Imagine the chaos that would have resulted if it had instead split ``int`` into two types: ``classic_int``, whose ``__div__`` implemented floor division, and ``new_int``, whose ``__div__`` implemented true division. This, in a more limited way, is the situation that Python number-crunchers currently find themselves in. In practice, the vast majority of projects have settled on the convention of using ``*`` for elementwise multiplication, and function call syntax for matrix multiplication (e.g., using ``numpy.ndarray`` instead of ``numpy.matrix``). This reduces the problems caused by API fragmentation, but it doesn't eliminate them. The strong desire to use infix notation for matrix multiplication has caused a number of specialized array libraries to continue to use the opposing convention (e.g., scipy.sparse, pyoperators, pyviennacl) despite the problems this causes, and ``numpy.matrix`` itself still gets used in introductory programming courses, often appears in StackOverflow answers, and so forth. Well-written libraries thus must continue to be prepared to deal with both types of objects, and, of course, are also stuck using unpleasant funcall syntax for matrix multiplication. After nearly two decades of trying, the numerical community has still not found any way to resolve these problems within the constraints of current Python syntax (see `Rejected alternatives to adding a new operator`_ below). This PEP proposes the minimum effective change to Python syntax that will allow us to drain this swamp. It splits ``*`` into two operators, just as was done for ``/``: ``*`` for elementwise multiplication, and ``@`` for matrix multiplication. (Why not the reverse? Because this way is compatible with the existing consensus, and because it gives us a consistent rule that all the built-in numeric operators also apply in an elementwise manner to arrays; the reverse convention would lead to more special cases.) So that's why matrix multiplication doesn't and can't just use ``*``. Now, in the the rest of this section, we'll explain why it nonetheless meets the high bar for adding a new operator. Why should matrix multiplication be infix? ------------------------------------------ Right now, most numerical code in Python uses syntax like ``numpy.dot(a, b)`` or ``a.dot(b)`` to perform matrix multiplication. This obviously works, so why do people make such a fuss about it, even to the point of creating API fragmentation and compatibility swamps? Matrix multiplication shares two features with ordinary arithmetic operations like addition and multiplication on numbers: (a) it is used very heavily in numerical programs -- often multiple times per line of code -- and (b) it has an ancient and universally adopted tradition of being written using infix syntax. This is because, for typical formulas, this notation is dramatically more readable than any function call syntax. Here's an example to demonstrate: One of the most useful tools for testing a statistical hypothesis is the linear hypothesis test for OLS regression models. It doesn't really matter what all those words I just said mean; if we find ourselves having to implement this thing, what we'll do is look up some textbook or paper on it, and encounter many mathematical formulas that look like: .. math:: S = (H \beta - r)^T (H V H^T)^{-1} (H \beta - r) Here the various variables are all vectors or matrices (details for the curious: [#lht]_). Now we need to write code to perform this calculation. In current numpy, matrix multiplication can be performed using either the function or method call syntax. Neither provides a particularly readable translation of the formula:: import numpy as np from numpy.linalg import inv, solve # Using dot function: S = np.dot((np.dot(H, beta) - r).T, np.dot(inv(np.dot(np.dot(H, V), H.T)), np.dot(H, beta) - r)) # Using dot method: S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r) With the ``@`` operator, the direct translation of the above formula becomes:: S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) Notice that there is now a transparent, 1-to-1 mapping between the symbols in the original formula and the code that implements it. Of course, an experienced programmer will probably notice that this is not the best way to compute this expression. The repeated computation of :math:`H \beta - r` should perhaps be factored out; and, expressions of the form ``dot(inv(A), B)`` should almost always be replaced by the more numerically stable ``solve(A, B)``. When using ``@``, performing these two refactorings gives us:: # Version 1 (as above) S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) # Version 2 trans_coef = H @ beta - r S = trans_coef.T @ inv(H @ V @ H.T) @ trans_coef # Version 3 S = trans_coef.T @ solve(H @ V @ H.T, trans_coef) Notice that when comparing between each pair of steps, it's very easy to see exactly what was changed. If we apply the equivalent transformations to the code using the .dot method, then the changes are much harder to read out or verify for correctness:: # Version 1 (as above) S = (H.dot(beta) - r).T.dot(inv(H.dot(V).dot(H.T))).dot(H.dot(beta) - r) # Version 2 trans_coef = H.dot(beta) - r S = trans_coef.T.dot(inv(H.dot(V).dot(H.T))).dot(trans_coef) # Version 3 S = trans_coef.T.dot(solve(H.dot(V).dot(H.T)), trans_coef) Readability counts! The statements using ``@`` are shorter, contain more whitespace, can be directly and easily compared both to each other and to the textbook formula, and contain only meaningful parentheses. This last point is particularly important for readability: when using function-call syntax, the required parentheses on every operation create visual clutter that makes it very difficult to parse out the overall structure of the formula by eye, even for a relatively simple formula like this one. Eyes are terrible at parsing non-regular languages. I made and caught many errors while trying to write out the 'dot' formulas above. I know they still contain at least one error, maybe more. (Exercise: find it. Or them.) The ``@`` examples, by contrast, are not only correct, they're obviously correct at a glance. If we are even more sophisticated programmers, and writing code that we expect to be reused, then considerations of speed or numerical accuracy might lead us to prefer some particular order of evaluation. Because ``@`` makes it possible to omit irrelevant parentheses, we can be certain that if we *do* write something like ``(H @ V) @ H.T``, then our readers will know that the parentheses must have been added intentionally to accomplish some meaningful purpose. In the ``dot`` examples, it's impossible to know which nesting decisions are important, and which are arbitrary. Infix ``@`` dramatically improves matrix code usability at all stages of programmer interaction. Transparent syntax is especially crucial for non-expert programmers ------------------------------------------------------------------- A large proportion of scientific code is written by people who are experts in their domain, but are not experts in programming. And there are many university courses run each year with titles like "Data analysis for social scientists" which assume no programming background, and teach some combination of mathematical techniques, introduction to programming, and the use of programming to implement these mathematical techniques, all within a 10-15 week period. These courses are more and more often being taught in Python rather than special-purpose languages like R or Matlab. For these kinds of users, whose programming knowledge is fragile, the existence of a transparent mapping between formulas and code often means the difference between succeeding and failing to write that code at all. This is so important that such classes often use the ``numpy.matrix`` type which defines ``*`` to mean matrix multiplication, even though this type is buggy and heavily disrecommended by the rest of the numpy community for the fragmentation that it causes. This pedagogical use case is, in fact, the *only* reason ``numpy.matrix`` remains a supported part of numpy. Adding ``@`` will benefit both beginning and advanced users with better syntax; and furthermore, it will allow both groups to standardize on the same notation from the start, providing a smoother on-ramp to expertise. But isn't matrix multiplication a pretty niche requirement? ----------------------------------------------------------- The world is full of continuous data, and computers are increasingly called upon to work with it in sophisticated ways. Arrays are the lingua franca of finance, machine learning, 3d graphics, computer vision, robotics, operations research, econometrics, meteorology, computational linguistics, recommendation systems, neuroscience, astronomy, bioinformatics (including genetics, cancer research, drug discovery, etc.), physics engines, quantum mechanics, geophysics, network analysis, and many other application areas. In most or all of these areas, Python is rapidly becoming a dominant player, in large part because of its ability to elegantly mix traditional discrete data structures (hash tables, strings, etc.) on an equal footing with modern numerical data types and algorithms. We all live in our own little sub-communities, so some Python users may be surprised to realize the sheer extent to which Python is used for number crunching -- especially since much of this particular sub-community's activity occurs outside of traditional Python/FOSS channels. So, to give some rough idea of just how many numerical Python programmers are actually out there, here are two numbers: In 2013, there were 7 international conferences organized specifically on numerical Python [#scipy-conf]_ [#pydata-conf]_. At PyCon 2014, ~20% of the tutorials appear to involve the use of matrices [#pycon-tutorials]_. To quantify this further, we used Github's "search" function to look at what modules are actually imported across a wide range of real-world code (i.e., all the code on Github). We checked for imports of several popular stdlib modules, a variety of numerically oriented modules, and various other extremely high-profile modules like django and lxml (the latter of which is the #1 most downloaded package on PyPI). Starred lines indicate packages which export array- or matrix-like objects which will adopt ``@`` if this PEP is approved:: Count of Python source files on Github matching given search terms (as of 2014-04-10, ~21:00 UTC) ================ ========== =============== ======= =========== module "import X" "from X import" total total/numpy ================ ========== =============== ======= =========== sys 2374638 63301 2437939 5.85 os 1971515 37571 2009086 4.82 re 1294651 8358 1303009 3.12 numpy ************** 337916 ********** 79065 * 416981 ******* 1.00 warnings 298195 73150 371345 0.89 subprocess 281290 63644 344934 0.83 django 62795 219302 282097 0.68 math 200084 81903 281987 0.68 threading 212302 45423 257725 0.62 pickle+cPickle 215349 22672 238021 0.57 matplotlib 119054 27859 146913 0.35 sqlalchemy 29842 82850 112692 0.27 pylab *************** 36754 ********** 41063 ** 77817 ******* 0.19 scipy *************** 40829 ********** 28263 ** 69092 ******* 0.17 lxml 19026 38061 57087 0.14 zlib 40486 6623 47109 0.11 multiprocessing 25247 19850 45097 0.11 requests 30896 560 31456 0.08 jinja2 8057 24047 32104 0.08 twisted 13858 6404 20262 0.05 gevent 11309 8529 19838 0.05 pandas ************** 14923 *********** 4005 ** 18928 ******* 0.05 sympy 2779 9537 12316 0.03 theano *************** 3654 *********** 1828 *** 5482 ******* 0.01 ================ ========== =============== ======= =========== These numbers should be taken with several grains of salt (see footnote for discussion: [#github-details]_), but, to the extent they can be trusted, they suggest that ``numpy`` might be the single most-imported non-stdlib module in the entire Pythonverse; it's even more-imported than such stdlib stalwarts as ``subprocess``, ``math``, ``pickle``, and ``threading``. And numpy users represent only a subset of the broader numerical community that will benefit from the ``@`` operator. Matrices may once have been a niche data type restricted to Fortran programs running in university labs and military clusters, but those days are long gone. Number crunching is a mainstream part of modern Python usage. In addition, there is some precedence for adding an infix operator to handle a more-specialized arithmetic operation: the floor division operator ``//``, like the bitwise operators, is very useful under certain circumstances when performing exact calculations on discrete values. But it seems likely that there are many Python programmers who have never had reason to use ``//`` (or, for that matter, the bitwise operators). ``@`` is no more niche than ``//``. So ``@`` is good for matrix formulas, but how common are those really? ---------------------------------------------------------------------- We've seen that ``@`` makes matrix formulas dramatically easier to work with for both experts and non-experts, that matrix formulas appear in many important applications, and that numerical libraries like numpy are used by a substantial proportion of Python's user base. But numerical libraries aren't just about matrix formulas, and being important doesn't necessarily mean taking up a lot of code: if matrix formulas only occured in one or two places in the average numerically-oriented project, then it still wouldn't be worth adding a new operator. So how common is matrix multiplication, really? When the going gets tough, the tough get empirical. To get a rough estimate of how useful the ``@`` operator will be, the table below shows the rate at which different Python operators are actually used in the stdlib, and also in two high-profile numerical packages -- the scikit-learn machine learning library, and the nipy neuroimaging library -- normalized by source lines of code (SLOC). Rows are sorted by the 'combined' column, which pools all three code bases together. The combined column is thus strongly weighted towards the stdlib, which is much larger than both projects put together (stdlib: 411575 SLOC, scikit-learn: 50924 SLOC, nipy: 37078 SLOC). [#sloc-details]_ The ``dot`` row (marked ``******``) counts how common matrix multiply operations are in each codebase. :: ==== ====== ============ ==== ======== op stdlib scikit-learn nipy combined ==== ====== ============ ==== ======== = 2969 5536 4932 3376 / 10,000 SLOC - 218 444 496 261 + 224 201 348 231 == 177 248 334 196 * 156 284 465 192 % 121 114 107 119 ** 59 111 118 68 != 40 56 74 44 / 18 121 183 41 > 29 70 110 39 += 34 61 67 39 < 32 62 76 38 >= 19 17 17 18 <= 18 27 12 18 dot ***** 0 ********** 99 ** 74 ****** 16 | 18 1 2 15 & 14 0 6 12 << 10 1 1 8 // 9 9 1 8 -= 5 21 14 8 *= 2 19 22 5 /= 0 23 16 4 >> 4 0 0 3 ^ 3 0 0 3 ~ 2 4 5 2 |= 3 0 0 2 &= 1 0 0 1 //= 1 0 0 1 ^= 1 0 0 0 **= 0 2 0 0 %= 0 0 0 0 <<= 0 0 0 0 >>= 0 0 0 0 ==== ====== ============ ==== ======== These two numerical packages alone contain ~780 uses of matrix multiplication. Within these packages, matrix multiplication is used more heavily than most comparison operators (``<`` ``!=`` ``<=`` ``>=``). Even when we dilute these counts by including the stdlib into our comparisons, matrix multiplication is still used more often in total than any of the bitwise operators, and 2x as often as ``//``. This is true even though the stdlib, which contains a fair amount of integer arithmetic and no matrix operations, makes up more than 80% of the combined code base. By coincidence, the numeric libraries make up approximately the same proportion of the 'combined' codebase as numeric tutorials make up of PyCon 2014's tutorial schedule, which suggests that the 'combined' column may not be *wildly* unrepresentative of new Python code in general. While it's impossible to know for certain, from this data it seems entirely possible that across all Python code currently being written, matrix multiplication is already used more often than ``//`` and the bitwise operations. But isn't it weird to add an operator with no stdlib uses? ---------------------------------------------------------- It's certainly unusual (though ``Ellipsis`` was also added without any stdlib uses). But the important thing is whether a change will benefit users, not where the software is being downloaded from. It's clear from the above that ``@`` will be used, and used heavily. And this PEP provides the critical piece that will allow the Python numerical community to finally reach consensus on a standard duck type for all array-like objects, which is a necessary precondition to ever adding a numerical array type to the stdlib. Matrix power and in-place operators ----------------------------------- The primary motivation for this PEP is ``@``; the other proposed operators don't have nearly as much impact. The matrix power operator ``@@`` is useful and well-defined, but not really necessary. It is still included, though, for consistency: if we have an ``@`` that is analogous to ``*``, then it would be weird and surprising to *not* have an ``@@`` that is analogous to ``**``. Similarly, the in-place operators ``@=`` and ``@@=`` provide limited value -- it's more common to write ``a = (b @ a)`` than it is to write ``a = (a @ b)``, and in-place matrix operations still generally have to allocate substantial temporary storage -- but they are included for completeness and symmetry. Compatibility considerations ============================ Currently, the only legal use of the ``@`` token in Python code is at statement beginning in decorators. The new operators are all infix; the one place they can never occur is at statement beginning. Therefore, no existing code will be broken by the addition of these operators, and there is no possible parsing ambiguity between decorator-@ and the new operators. Another important kind of compatibility is the mental cost paid by users to update their understanding of the Python language after this change, particularly for users who do not work with matrices and thus do not benefit. Here again, ``@`` has minimal impact: even comprehensive tutorials and references will only need to add a sentence or two to fully document this PEP's changes for a non-numerical audience. Intended usage details ====================== This section is informative, rather than normative -- it documents the consensus of a number of libraries that provide array- or matrix-like objects on how the ``@`` and ``@@`` operators will be implemented. This section uses the numpy terminology for describing arbitrary multidimensional arrays of data, because it is a superset of all other commonly used models. In this model, the *shape* of any array is represented by a tuple of integers. Because matrices are two-dimensional, they have len(shape) == 2, while 1d vectors have len(shape) == 1, and scalars have shape == (), i.e., they are "0 dimensional". Any array contains prod(shape) total entries. Notice that `prod(()) == 1`_ (for the same reason that sum(()) == 0); scalars are just an ordinary kind of array, not a special case. Notice also that we distinguish between a single scalar value (shape == (), analogous to ``1``), a vector containing only a single entry (shape == (1,), analogous to ``[1]``), a matrix containing only a single entry (shape == (1, 1), analogous to ``[[1]]``), etc., so the dimensionality of any array is always well-defined. Other libraries with more restricted representations (e.g., those that support 2d arrays only) might implement only a subset of the functionality described here. .. _prod(()) == 1: https://en.wikipedia.org/wiki/Empty_product Semantics --------- The recommended semantics for ``@`` for different inputs are: * 2d inputs are conventional matrices, and so the semantics are obvious: we apply conventional matrix multiplication. If we write ``arr(2, 3)`` to represent an arbitrary 2x3 array, then ``arr(3, 4) @ arr(4, 5)`` returns an array with shape (3, 5). * 1d vector inputs are promoted to 2d by prepending or appending a '1' to the shape, the operation is performed, and then the added dimension is removed from the output. The 1 is always added on the "outside" of the shape: prepended for left arguments, and appended for right arguments. The result is that matrix @ vector and vector @ matrix are both legal (assuming compatible shapes), and both return 1d vectors; vector @ vector returns a scalar. This is clearer with examples. * ``arr(2, 3) @ arr(3, 1)`` is a regular matrix product, and returns an array with shape (2, 1), i.e., a column vector. * ``arr(2, 3) @ arr(3)`` performs the same computation as the previous (i.e., treats the 1d vector as a matrix containing a single *column*, shape = (3, 1)), but returns the result with shape (2,), i.e., a 1d vector. * ``arr(1, 3) @ arr(3, 2)`` is a regular matrix product, and returns an array with shape (1, 2), i.e., a row vector. * ``arr(3) @ arr(3, 2)`` performs the same computation as the previous (i.e., treats the 1d vector as a matrix containing a single *row*, shape = (1, 3)), but returns the result with shape (2,), i.e., a 1d vector. * ``arr(1, 3) @ arr(3, 1)`` is a regular matrix product, and returns an array with shape (1, 1), i.e., a single value in matrix form. * ``arr(3) @ arr(3)`` performs the same computation as the previous, but returns the result with shape (), i.e., a single scalar value, not in matrix form. So this is the standard inner product on vectors. An infelicity of this definition for 1d vectors is that it makes ``@`` non-associative in some cases (``(Mat1 @ vec) @ Mat2`` != ``Mat1 @ (vec @ Mat2)``). But this seems to be a case where practicality beats purity: non-associativity only arises for strange expressions that would never be written in practice; if they are written anyway then there is a consistent rule for understanding what will happen (``Mat1 @ vec @ Mat2`` is parsed as ``(Mat1 @ vec) @ Mat2``, just like ``a - b - c``); and, not supporting 1d vectors would rule out many important use cases that do arise very commonly in practice. No-one wants to explain to new users why to solve the simplest linear system in the obvious way, they have to type ``(inv(A) @ b[:, np.newaxis]).flatten()`` instead of ``inv(A) @ b``, or perform an ordinary least-squares regression by typing ``solve(X.T @ X, X @ y[:, np.newaxis]).flatten()`` instead of ``solve(X.T @ X, X @ y)``. No-one wants to type ``(a[np.newaxis, :] @ b[:, np.newaxis])[0, 0]`` instead of ``a @ b`` every time they compute an inner product, or ``(a[np.newaxis, :] @ Mat @ b[:, np.newaxis])[0, 0]`` for general quadratic forms instead of ``a @ Mat @ b``. In addition, sage and sympy (see below) use these non-associative semantics with an infix matrix multiplication operator (they use ``*``), and they report that they haven't experienced any problems caused by it. * For inputs with more than 2 dimensions, we treat the last two dimensions as being the dimensions of the matrices to multiply, and 'broadcast' across the other dimensions. This provides a convenient way to quickly compute many matrix products in a single operation. For example, ``arr(10, 2, 3) @ arr(10, 3, 4)`` performs 10 separate matrix multiplies, each of which multiplies a 2x3 and a 3x4 matrix to produce a 2x4 matrix, and then returns the 10 resulting matrices together in an array with shape (10, 2, 4). The intuition here is that we treat these 3d arrays of numbers as if they were 1d arrays *of matrices*, and then apply matrix multiplication in an elementwise manner, where now each 'element' is a whole matrix. Note that broadcasting is not limited to perfectly aligned arrays; in more complicated cases, it allows several simple but powerful tricks for controlling how arrays are aligned with each other; see [#broadcasting]_ for details. (In particular, it turns out that when broadcasting is taken into account, the standard scalar * matrix product is a special case of the elementwise multiplication operator ``*``.) If one operand is >2d, and another operand is 1d, then the above rules apply unchanged, with 1d->2d promotion performed before broadcasting. E.g., ``arr(10, 2, 3) @ arr(3)`` first promotes to ``arr(10, 2, 3) @ arr(3, 1)``, then broadcasts the right argument to create the aligned operation ``arr(10, 2, 3) @ arr(10, 3, 1)``, multiplies to get an array with shape (10, 2, 1), and finally removes the added dimension, returning an array with shape (10, 2). Similarly, ``arr(2) @ arr(10, 2, 3)`` produces an intermediate array with shape (10, 1, 3), and a final array with shape (10, 3). * 0d (scalar) inputs raise an error. Scalar * matrix multiplication is a mathematically and algorithmically distinct operation from matrix @ matrix multiplication, and is already covered by the elementwise ``*`` operator. Allowing scalar @ matrix would thus both require an unnecessary special case, and violate TOOWTDI. The recommended semantics for ``@@`` are:: def __matpow__(self, n): if not isinstance(n, numbers.Integral): raise TypeError("@@ not implemented for fractional powers") if n == 0: return identity_matrix_with_shape(self.shape) elif n < 0: return inverse(self) @ (self @@ (n + 1)) else: return self @ (self @@ (n - 1)) (Of course we expect that much more efficient implementations will be used in practice.) Notice that if given an appropriate definition of ``identity_matrix_with_shape``, then this definition will automatically handle >2d arrays appropriately. Notice also that with this definition, ``vector @@ 2`` gives the squared Euclidean length of the vector, a commonly used value. Also, while it is rarely useful to explicitly compute inverses or other negative powers in standard immediate-mode dense matrix code, these computations are natural when doing symbolic or deferred-mode computations (as in e.g. sympy, theano, numba, numexpr); therefore, negative powers are fully supported. Fractional powers, though, bring in variety of `mathematical complications`_, so we leave it to individual projects to decide whether they want to try to define some reasonable semantics for fractional inputs. .. _`mathematical complications`: https://en.wikipedia.org/wiki/Square_root_of_a_matrix Adoption -------- We group existing Python projects which provide array- or matrix-like types based on what API they currently use for elementwise and matrix multiplication. **Projects which currently use * for *elementwise* multiplication, and function/method calls for *matrix* multiplication:** The developers of the following projects have expressed an intention to implement ``@`` and ``@@`` on their array-like types using the above semantics: * numpy * pandas * blaze * theano The following projects have been alerted to the existence of the PEP, but it's not yet known what they plan to do if it's accepted. We don't anticipate that they'll have any objections, though, since everything proposed here is consistent with how they already do things: * pycuda * panda3d **Projects which currently use * for *matrix* multiplication, and function/method calls for *elementwise* multiplication:** The following projects have expressed an intention, if this PEP is accepted, to migrate from their current API to the elementwise-``*``, matmul-``@`` convention (i.e., this is a list of projects whose API fragmentation will probably be eliminated if this PEP is accepted): * numpy (``numpy.matrix``) * scipy.sparse * pyoperators * pyviennacl The following projects have been alerted to the existence of the PEP, but it's not known what they plan to do if it's accepted (i.e., this is a list of projects whose API fragmentation may or may not be eliminated if this PEP is accepted): * cvxopt **Projects which currently use * for *matrix* multiplication, and which do not implement elementwise multiplication at all:** There are several projects which implement matrix types, but from a very different perspective than the numerical libraries discussed above. These projects focus on computational methods for analyzing matrices in the sense of abstract mathematical objects (i.e., linear maps over free modules over rings), rather than as big bags full of numbers that need crunching. And it turns out that from the abstract math point of view, there isn't much use for elementwise operations in the first place; as discussed in the Background section above, elementwise operations are motivated by the bag-of-numbers approach. The different goals of these projects mean that they don't encounter the basic problem that this PEP exists to address, making it mostly irrelevant to them; while they appear superficially similar, they're actually doing something quite different. They use ``*`` for matrix multiplication (and for group actions, and so forth), and if this PEP is accepted, their expressed intention is to continue doing so, while perhaps adding ``@`` and ``@@`` on matrices as aliases for ``*`` and ``**``: * sympy * sage If you know of any actively maintained Python libraries which provide an interface for working with numerical arrays or matrices, and which are not listed above, then please let the PEP author know: njs(a)pobox.com Rationale for specification details =================================== Choice of operator ------------------ Why ``@`` instead of some other punctuation symbol? It doesn't matter much, and there isn't any consensus across other programming languages about how this operator should be named [#matmul-other-langs]_, but ``@`` has a few advantages: * ``@`` is a friendly character that Pythoneers are already used to typing in decorators, and its use in email addresses means it is more likely to be easily accessible across keyboard layouts than some other characters (e.g. ``$`` or non-ASCII characters). * The mATrices mnemonic is cute. * It's round like ``*`` and :math:`\cdot`. * The use of a single-character token makes ``@@`` possible, which is a nice bonus. * The swirly shape is reminiscent of the simultaneous sweeps over rows and columns that define matrix multiplication; its asymmetry is evocative of its non-commutative nature. (Non)-Definitions for built-in types ------------------------------------ No ``__matmul__`` or ``__matpow__`` are defined for builtin numeric types (``float``, ``int``, etc.) or for the ``numbers.Number`` hierarchy, because these types represent scalars, and the consensus semantics for ``@`` are that it should raise an error on scalars. We do not -- for now -- define a ``__matmul__`` method on the standard ``memoryview`` or ``array.array`` objects, for several reasons. First, there is currently no way to create multidimensional memoryview objects using only the stdlib, and array objects cannot represent multidimensional data at all, which makes ``__matmul__`` much less useful. Second, providing a quality implementation of matrix multiplication is highly non-trivial. Naive nested loop implementations are very slow and providing one in CPython would just create a trap for users. But the alternative -- providing a modern, competitive matrix multiply -- would require that CPython link to a BLAS library, which brings a set of new complications. In particular, several popular BLAS libraries (including the one that ships by default on OS X) currently break the use of ``multiprocessing`` [#blas-fork]_. And finally, we'd have to add quite a bit beyond ``__matmul__`` before ``memoryview`` or ``array.array`` would be useful for numeric work -- like elementwise versions of the other arithmetic operators, just to start. Put together, these considerations mean that the cost/benefit of adding ``__matmul__`` to these types just isn't there, so for now we'll continue to delegate these problems to numpy and friends, and defer a more systematic solution to a future proposal. There are also non-numeric Python builtins which define ``__mul__`` (``str``, ``list``, ...). We do not define ``__matmul__`` for these types either, because why would we even do that. Unresolved issues ----------------- Associativity of ``@`` '''''''''''''''''''''' It's been suggested that ``@`` should be right-associative, on the grounds that for expressions like ``Mat @ Mat @ vec``, the two different evaluation orders produce the same result, but the right-associative order ``Mat @ (Mat @ vec)`` will be faster and use less memory than the left-associative order ``(Mat @ Mat) @ vec``. (Matrix-vector multiplication is much cheaper than matrix-matrix multiplication). It would be a shame if users found themselves required to use an overabundance of parentheses to achieve acceptable speed/memory usage in common situations, but, it's not currently clear whether such cases actually are common enough to override Python's general rule of left-associativity, or even whether they're more common than the symmetric cases where left-associativity would be faster (though this does seem intuitively plausible). The only way to answer this is probably to do an audit of some real-world uses and check how often the associativity matters in practice; if this PEP is accepted in principle, then we should probably do this check before finalizing it. Rejected alternatives to adding a new operator ============================================== Over the past few decades, the Python numeric community has explored a variety of ways to resolve the tension between matrix and elementwise multiplication operations. PEP 211 and PEP 225, both proposed in 2000 and last seriously discussed in 2008 [#threads-2008]_, were early attempts to add new operators to solve this problem, but suffered from serious flaws; in particular, at that time the Python numerical community had not yet reached consensus on the proper API for array objects, or on what operators might be needed or useful (e.g., PEP 225 proposes 6 new operators with unspecified semantics). Experience since then has now led to consensus that the best solution, for both numeric Python and core Python, is to add a single infix operator for matrix multiply (together with the other new operators this implies like ``@=``). We review some of the rejected alternatives here. **Use a second type that defines __mul__ as matrix multiplication:** As discussed above (`Background: What's wrong with the status quo?`_), this has been tried this for many years via the ``numpy.matrix`` type (and its predecessors in Numeric and numarray). The result is a strong consensus among both numpy developers and developers of downstream packages that ``numpy.matrix`` should essentially never be used, because of the problems caused by having conflicting duck types for arrays. (Of course one could then argue we should *only* define ``__mul__`` to be matrix multiplication, but then we'd have the same problem with elementwise multiplication.) There have been several pushes to remove ``numpy.matrix`` entirely; the only counter-arguments have come from educators who find that its problems are outweighed by the need to provide a simple and clear mapping between mathematical notation and code for novices (see `Transparent syntax is especially crucial for non-expert programmers`_). But, of course, starting out newbies with a dispreferred syntax and then expecting them to transition later causes its own problems. The two-type solution is worse than the disease. **Add lots of new operators, or add a new generic syntax for defining infix operators:** In addition to being generally un-Pythonic and repeatedly rejected by BDFL fiat, this would be using a sledgehammer to smash a fly. The scientific python community has consensus that adding one operator for matrix multiplication is enough to fix the one otherwise unfixable pain point. (In retrospect, we all think PEP 225 was a bad idea too -- or at least far more complex than it needed to be.) **Add a new @ (or whatever) operator that has some other meaning in general Python, and then overload it in numeric code:** This was the approach taken by PEP 211, which proposed defining ``@`` to be the equivalent of ``itertools.product``. The problem with this is that when taken on its own terms, adding an infix operator for ``itertools.product`` is just silly. (During discussions of this PEP, a similar suggestion was made to define ``@`` as a general purpose function composition operator, and this suffers from the same problem; ``functools.compose`` isn't even useful enough to exist.) Matrix multiplication has a uniquely strong rationale for inclusion as an infix operator. There almost certainly don't exist any other binary operations that will ever justify adding any other infix operators to Python. **Add a .dot method to array types so as to allow "pseudo-infix" A.dot(B) syntax:** This has been in numpy for some years, and in many cases it's better than dot(A, B). But it's still much less readable than real infix notation, and in particular still suffers from an extreme overabundance of parentheses. See `Why should matrix multiplication be infix?`_ above. **Use a 'with' block to toggle the meaning of * within a single code block**: E.g., numpy could define a special context object so that we'd have:: c = a * b # element-wise multiplication with numpy.mul_as_dot: c = a * b # matrix multiplication However, this has two serious problems: first, it requires that every array-like type's ``__mul__`` method know how to check some global state (``numpy.mul_is_currently_dot`` or whatever). This is fine if ``a`` and ``b`` are numpy objects, but the world contains many non-numpy array-like objects. So this either requires non-local coupling -- every numpy competitor library has to import numpy and then check ``numpy.mul_is_currently_dot`` on every operation -- or else it breaks duck-typing, with the above code doing radically different things depending on whether ``a`` and ``b`` are numpy objects or some other sort of object. Second, and worse, ``with`` blocks are dynamically scoped, not lexically scoped; i.e., any function that gets called inside the ``with`` block will suddenly find itself executing inside the mul_as_dot world, and crash and burn horribly -- if you're lucky. So this is a construct that could only be used safely in rather limited cases (no function calls), and which would make it very easy to shoot yourself in the foot without warning. **Use a language preprocessor that adds extra numerically-oriented operators and perhaps other syntax:** (As per recent BDFL suggestion: [#preprocessor]_) This suggestion seems based on the idea that numerical code needs a wide variety of syntax additions. In fact, given ``@``, most numerical users don't need any other operators or syntax; it solves the one really painful problem that cannot be solved by other means, and that causes painful reverberations through the larger ecosystem. Defining a new language (presumably with its own parser which would have to be kept in sync with Python's, etc.), just to support a single binary operator, is neither practical nor desireable. In the numerical context, Python's competition is special-purpose numerical languages (Matlab, R, IDL, etc.). Compared to these, Python's killer feature is exactly that one can mix specialized numerical code with code for XML parsing, web page generation, database access, network programming, GUI libraries, and so forth, and we also gain major benefits from the huge variety of tutorials, reference material, introductory classes, etc., which use Python. Fragmenting "numerical Python" from "real Python" would be a major source of confusion. A major motivation for this PEP is to *reduce* fragmentation. Having to set up a preprocessor would be an especially prohibitive complication for unsophisticated users. And we use Python because we like Python! We don't want almost-but-not-quite-Python. **Use overloading hacks to define a "new infix operator" like *dot*, as in a well-known Python recipe:** (See: [#infix-hack]_) Beautiful is better than ugly. This is... not beautiful. And not Pythonic. And especially unfriendly to beginners, who are just trying to wrap their heads around the idea that there's a coherent underlying system behind these magic incantations that they're learning, when along comes an evil hack like this that violates that system, creates bizarre error messages when accidentally misused, and whose underlying mechanisms can't be understood without deep knowledge of how object oriented systems work. We've considered promoting this as a general solution, and perhaps if the PEP is rejected we'll revisit this option, but so far the numeric community has mostly elected to leave this one on the shelf. References ========== .. [#preprocessor] From a comment by GvR on a G+ post by GvR; the comment itself does not seem to be directly linkable: https://plus.google.com/115212051037621986145/posts/hZVVtJ9bK3u .. [#infix-hack] http://code.activestate.com/recipes/384122-infix-operators/ http://www.sagemath.org/doc/reference/misc/sage/misc/decorators.html#sage.m… .. [#scipy-conf] http://conference.scipy.org/past.html .. [#pydata-conf] http://pydata.org/events/ .. [#lht] In this formula, :math:`\beta` is a vector or matrix of regression coefficients, :math:`V` is the estimated variance/covariance matrix for these coefficients, and we want to test the null hypothesis that :math:`H\beta = r`; a large :math:`S` then indicates that this hypothesis is unlikely to be true. For example, in an analysis of human height, the vector :math:`\beta` might contain one value which was the the average height of the measured men, and another value which was the average height of the measured women, and then setting :math:`H = [1, -1], r = 0` would let us test whether men and women are the same height on average. Compare to eq. 2.139 in http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/tutorials/xegbohtmlnod… Example code is adapted from https://github.com/rerpy/rerpy/blob/0d274f85e14c3b1625acb22aed1efa85d122ecb… .. [#pycon-tutorials] Out of the 36 tutorials scheduled for PyCon 2014 (https://us.pycon.org/2014/schedule/tutorials/), we guess that the 8 below will almost certainly deal with matrices: * Dynamics and control with Python * Exploring machine learning with Scikit-learn * How to formulate a (science) problem and analyze it using Python code * Diving deeper into Machine Learning with Scikit-learn * Data Wrangling for Kaggle Data Science Competitions – An etude * Hands-on with Pydata: how to build a minimal recommendation engine. * Python for Social Scientists * Bayesian statistics made simple In addition, the following tutorials could easily involve matrices: * Introduction to game programming * mrjob: Snakes on a Hadoop *("We'll introduce some data science concepts, such as user-user similarity, and show how to calculate these metrics...")* * Mining Social Web APIs with IPython Notebook * Beyond Defaults: Creating Polished Visualizations Using Matplotlib This gives an estimated range of 8 to 12 / 36 = 22% to 33% of tutorials dealing with matrices; saying ~20% then gives us some wiggle room in case our estimates are high. .. [#sloc-details] SLOCs were defined as physical lines which contain at least one token that is not a COMMENT, NEWLINE, ENCODING, INDENT, or DEDENT. Counts were made by using ``tokenize`` module from Python 3.2.3 to examine the tokens in all files ending ``.py`` underneath some directory. Only tokens which occur at least once in the source trees are included in the table. The counting script will be available as an auxiliary file once this PEP is submitted; until then, it can be found here: https://gist.github.com/njsmith/9157645 Matrix multiply counts were estimated by counting how often certain tokens which are used as matrix multiply function names occurred in each package. In principle this could create false positives, but as far as I know the counts are exact; it's unlikely that anyone is using ``dot`` as a variable name when it's also the name of one of the most widely-used numpy functions. All counts were made using the latest development version of each project as of 21 Feb 2014. 'stdlib' is the contents of the Lib/ directory in commit d6aa3fa646e2 to the cpython hg repository, and treats the following tokens as indicating matrix multiply: n/a. 'scikit-learn' is the contents of the sklearn/ directory in commit 69b71623273ccfc1181ea83d8fb9e05ae96f57c7 to the scikit-learn repository (https://github.com/scikit-learn/scikit-learn), and treats the following tokens as indicating matrix multiply: ``dot``, ``fast_dot``, ``safe_sparse_dot``. 'nipy' is the contents of the nipy/ directory in commit 5419911e99546401b5a13bd8ccc3ad97f0d31037 to the nipy repository (https://github.com/nipy/nipy/), and treats the following tokens as indicating matrix multiply: ``dot``. .. [#blas-fork] BLAS libraries have a habit of secretly spawning threads, even when used from single-threaded programs. And threads play very poorly with ``fork()``; the usual symptom is that attempting to perform linear algebra in a child process causes an immediate deadlock. .. [#threads-2008] http://fperez.org/py4science/numpy-pep225/numpy-pep225.html .. [#broadcasting] http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html .. [#matmul-other-langs] http://mail.scipy.org/pipermail/scipy-user/2014-February/035499.html .. [#github-details] Counts were produced by manually entering the string ``"import foo"`` or ``"from foo import"`` (with quotes) into the Github code search page, e.g.: https://github.com/search?q=%22import+numpy%22&ref=simplesearch&type=Code on 2014-04-10 at ~21:00 UTC. The reported values are the numbers given in the "Languages" box on the lower-left corner, next to "Python". This also causes some undercounting (e.g., leaving out Cython code, and possibly one should also count HTML docs and so forth), but these effects are negligible (e.g., only ~1% of numpy usage appears to occur in Cython code, and probably even less for the other modules listed). The use of this box is crucial, however, because these counts appear to be stable, while the "overall" counts listed at the top of the page ("We've found ___ code results") are highly variable even for a single search -- simply reloading the page can cause this number to vary by a factor of 2 (!!). (They do seem to settle down if one reloads the page repeatedly, but nonetheless this is spooky enough that it seemed better to avoid these numbers.) These numbers should of course be taken with multiple grains of salt; it's not clear how representative Github is of Python code in general, and limitations of the search tool make it impossible to get precise counts. AFAIK this is the best data set currently available, but it'd be nice if it were better. In particular: * Lines like ``import sys, os`` will only be counted in the ``sys`` row. * A file containing both ``import X`` and ``from X import`` will be counted twice * Imports of the form ``from X.foo import ...`` are missed. We could catch these by instead searching for "from X", but this is a common phrase in English prose, so we'd end up with false positives from comments, strings, etc. For many of the modules considered this shouldn't matter too much -- for example, the stdlib modules have flat namespaces -- but it might especially lead to undercounting of django, scipy, and twisted. Also, it's possible there exist other non-stdlib modules we didn't think to test that are even more-imported than numpy -- though we tried quite a few of the obvious suspects. If you find one, let us know! The modules tested here were chosen based on a combination of intuition and the top-100 list at pypi-ranking.info. Fortunately, it doesn't really matter if it turns out that numpy is, say, merely the *third* most-imported non-stdlib module, since the point is just that numeric programming is a common and mainstream activity. Finally, we should point out the obvious: whether a package is import**ed** is rather different from whether it's import**ant**. No-one's claiming numpy is "the most important package" or anything like that. Certainly more packages depend on distutils, e.g., then depend on numpy -- and far fewer source files import distutils than import numpy. But this is fine for our present purposes. Most source files don't import distutils because most source files don't care how they're distributed, so long as they are; these source files thus don't care about details of how distutils' API works. This PEP is in some sense about changing how numpy's and related packages' APIs work, so the relevant metric is to look at source files that are choosing to directly interact with that API, which is sort of like what we get by looking at import statements. .. [#hugunin] The first such proposal occurs in Jim Hugunin's very first email to the matrix SIG in 1995, which lays out the first draft of what became Numeric. He suggests using ``*`` for elementwise multiplication, and ``%`` for matrix multiplication: https://mail.python.org/pipermail/matrix-sig/1995-August/000002.html Copyright ========= This document has been placed in the public domain. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

36 155

Preserving **kwargs order (was: Re: OrderedDict literals)
by Eric Snow April 3, 2014

April 3, 2014

On Wed, Mar 19, 2014 at 4:01 PM, <random832(a)fastmail.us> wrote: > On Wed, Mar 19, 2014, at 17:24, Eric Snow wrote: >> FYI I'm working on a proposal to support preserving the call order of >> **kwargs. This would allow initializing an OrderedDict correctly >> using keyword args. I may have something to show for it by the end of >> the PyCon sprints. > > I know that it's possible to support this* in dict, but is kwargs > required to be a dict? I'm not talking about the ** unpacking syntax, but about the function definition syntax for **kwargs: def spam(a, b, **kwargs): ... > What route are you going: modify dict to allow > this and require dict to support it; make a new class for kwargs and > require this behavior of kwargs; modify dict on CPython to allow it but > only require that kwargs support it "somehow" on non-CPython? Hopefully I'll have time to write a proto-PEP on this in the next couple weeks, but the gist is that I see 2 options: 1. Change kwargs to an OrderedDict for functions decorated a special decorator: @preserve_kwargs_order def spam(a, b, **kwargs): ... # kwargs will be an OrderedDict 2. Store the ordered keys in a list and bind that to a special local variable, but only for functions that have **kwargs: def spam(a, b, **kwargs): kwargs = OrderedDict((k, kwargs[k]) for k in __kwargs_order__) ... My gut feeling is that option 2 is the better choice. And before you ask, Guido has already vetoed making **kwargs always be an OrderedDict. There are a few other details to work out, but the above is the main thing. Also, an OrderedDict implemented in C will likely be a prerequisite, but luckily I've had a patch up since July(?) (#16991). It doesn't apply cleanly at present, but I expect that's only barely and I should be able to get a clean patch up when I have a chance. There wasn't a lot of interest in reviewing such a large patch so I'd back-burnered it until Python 3.4 got released (which just happened). Once I get a couple reviews I should be able to get that in (wink wink nudge nudge). -eric

8 16

Fixing the Python 3 bytes constructor
by Nick Coghlan April 3, 2014

April 3, 2014

One of the current annoyances with the bytes type in Python 3 is the way the constructor handles integers: >>> bytes(3) b'\x00\x00\x00' It would be far more consistent with the behaviour of other bytes interfaces if the result of that call was instead b'\x03'. Instead, to get that behaviour, you currently have to wrap it in a list or other iterable: >>> bytes([3]) b'\x03' The other consequence of this is that's currently no neat way to convert the integers produced by various bytes APIs back to a length one bytes object - we have no binary equivalent of "chr" to convert an integer in the range 0-255 inclusive to its bytes counterpart. The acceptance of PEP 361 means we'll get another option (b"%c".__mod__) but that's hardly what anyone would call obvious. However, during a conversation today, a possible solution occurred to me: a "bytes.chr" class method, that served as an alternate constructor. That idea results in the following 3 part proposal: 1. Add "bytes.chr" such that "bytes.chr(x)" is equivalent to the PEP 361 defined "b'%c' % x" 2. Add "bytearray.allnull" and "bytes.allnull" such that "bytearray.allnull(x)" is equivalent to the current "bytearray(x)" int handling 3. Deprecate the current "bytes(x)" and "bytearray(x)" int handling as not only ambiguous, but actually a genuine bug magnet (it's way too easy to accidentally pass a large integer and try to allocate a ridiculously large bytes object) For point 2, I also considered the following alternative names before settling on "allnull": - bytes.null sounds too much like an alias for b"\x00" - bytes.nulls just sounded too awkward to say (too many sibilants) - bytes.zeros I can never remember how to spell (bytes.zeroes?) - bytearray.cleared sort of worked, but bytes.cleared? - ditto for bytearray.prealloc and bytes.prealloc (latter makes no sense) That last is also a very C-ish name (although it is a rather C-ish operation). Anyway, what do people think? Does anyone actually *like* the way the bytes constructor in Python 3 currently handles integers and want to keep it forever? Does the above proposal sound like a reasonable suggestion for improvement in 3.5? Does this hit PEP territory, since it's changing the signature and API of a builtin? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

27 73

Browser for mailing lists
by Richard Prosser April 1, 2014

April 1, 2014

I wanted to search this list (and other Python ones) but it is practically impossible to do so. So why not add a tool like http://markmail.org/search/?q=python to https://mail.python.org/mailman/listinfo/python-ideas or the like? I got this from http://openmeetings.apache.org/mail-lists.html Cheers, Richard

18 38

Making sys.std* thread-local
by Antony Lee March 31, 2014

March 31, 2014

The recent discussion about lexical vs. dynamic scoping reminded me of the following issue, that plagues e.g. the new contextlib.redirect_stdout context manager: the standard file-like objects sys.std* are global instead of thread-local, making their manipulation unsafe in multithreaded programs (this issue is related to lexical vs dynamic scoping because in a language with (optional) dynamic scoping, these objects can be made dynamically scoped, thus solving the issue). Of course, changing sys.std* to being thread-local would be backwards incompatible, but perhaps some new objects, e.g. sys.thread_local_std*, could be added to sys, with print() and related functions using sys.thread_local_std* if it is set and sys.std* otherwise. Antony

3 3

Fw: Allow specifying list of functions to `map`
by Andrew Barnert March 30, 2014

March 30, 2014

Sorry, google-grouped, so I'm not sure if this went out to the list or not; apologies if it's a dup… From: Ram Rachum <ram.rachum(a)gmail.com> Sent: Sunday, March 30, 2014 10:59 AM > If `map` accepted a tuple for its first argument, things like this: > > stdout, stderr = map(str.strip, map(bytes.decode, popen.communicate())) > > Could become this: > > stdout, stderr = map((bytes.decode, str.strip), popen.communicate()) I'm not sure whether to read that as returning: map(bytes.decode, x), map(str.strip, x) map(compose(bytes.decode, str.strip), x) map(rcompose(bytes.decode, str.strip), x) None of the functional languages that Python borrowed map from have any of these, but that's because they all have compose as a builtin, or even an operator: map(bytes.decode @ str.strip, popen.communicate()) But I think in Python this is much more readable as an expression on each value rather than a higher-order function: (text.decode().strip() for text in popen.communicate()) Even in Haskell, a listcomp is often more readable than mapping a compose or other higher-order function, but that's much more true when you're using Python-style method calls.

1 0

Allow specifying list of functions to `map`
by Ram Rachum March 30, 2014

March 30, 2014

If `map` accepted a tuple for its first argument, things like this: stdout, stderr = map(str.strip, map(bytes.decode, popen.communicate())) Could become this: stdout, stderr = map((bytes.decode, str.strip), popen.communicate())

3 3

Re: [Python-ideas] for-loop-if like list comps have?
by Christopher Welborn March 30, 2014

March 30, 2014

On 03/30/2014 05:44 AM, Liam Marsh wrote: > hello, > is it possible to accept this syntax? > it is about the same subject, but an other proposition... > so, here it is: > > >>>for x in xs , y in ys: > ... � �***instructions*** > thank you! > > I think you meant to send this to the list, instead of my personal email. I use thunderbird and news.gmane.org to send things to python-ideas, but I think you can just send a mail to: Python-ideas(a)python.org with 'Re: <the original subject line>' such as: Re: [Python-ideas] for-loop-if like list comps have? -- Christopher Welborn (cj) cjwelborn at live.com http://welbornprod.com

2 1

Nested dictionaries
by Richard Prosser March 29, 2014

March 29, 2014

Not the first time I found myself recently needing to create "multi-dimensional" dictionaries, so that I could write something like: my_dict = ['a']['b']['b']['a'] = 'fred' and once again I wonder why Python does not support that idiom directly, as it must be a fairly common requirement. I know that there are good solutions like the one at http://ohuiginn.net/mt/2010/07/nested_dictionaries_in_python.html or by using defaultdict but I feel that this really should be part of the language. So, should I propose a PEP? Thanks ... Richard

7 6

Re: [Python-ideas] Nested dictionaries
by Masklinn March 29, 2014

March 29, 2014

On 2014-03-29, at 20:43 , Richard Prosser <richard.prosser(a)mail.com> wrote: > Maybe so but it is not straightforward and therefore IMO not "Pythonic". I don't believe autovivification to be pythonic in the first place[0]. Python making you roll your own thus feels sensible. > I really don't understand the objections that are being raised. What is wrong with being able to use a simple expression? > > > Richard > PS Thanks for the responses everyone but it would nice if someone agreed with my proposal! [0] Steven already mentioned "flat is better than nested", but "explicit is better than implicit" also applies. To the surprise of no one, using autovivification is idiomatic perl.

1 0

Persisting private function state between calls (was: "Read Only" namespaces)
by Eric Snow March 29, 2014

March 29, 2014

On Sat, Mar 29, 2014 at 9:51 AM, Richard Prosser <richard.prosser(a)mail.com> wrote: > Quite simply, the principle is to allow changes to variables within a > function body (for example) but not outside of it, rather like the > Functional Programming paradigm. That of course excludes the use of global > variables and requires functions to return whatever is needed to the outer > scope. In my experience, globals are only needed to implement persistent > (inter-call) states anyway, so assuming that such a feature is available in > the language I believe that the "Read Only" proposal is viable. To restate, you want a way to have some variables in your function body that persist between calls without using global variables. There are already several ways to do this, though none is particularly obvious. Let's use a simple example using a global variable followed by the alternative solutions. current = 0 def adder(x): global current current += x return current 1. Still use a global variable, but make it "private" (just use the convention of a leading underscore on the name): _current = 0 def adder(x): global _current _current += x return _current 2. Use the "default argument hack": def adder(x, _ns={'current': 0}): _ns['current'] += x return _ns['current'] or class IntWrapper: def __init__(self, value=0): self.value = value def adder(x, _current=IntWrapper()): _current.value += x return _current.value 3. Use a closure (making use of the "nonlocal" keyword): def get_adder(start=0): current = start # Done for clarity. def adder(x): nonlocal current current += x return current return adder adder = get_adder() Each of the four approaches has its pros and cons. The closure approach is probably the clearest if you really want to avoid using a global. If there were agreement on a best approach, the least we could do it make it clearer in the docs, assuming you looked and didn't find anything helpful in the obvious places. :) A more explicit mechanism could be nice, but would have to pass a pretty high bar for inclusion into the language. Here are some approaches: * A new keyword, e.g. "persistent", to accompany global and nonlocal. * A special decorator that identifies persistent local variables in the function. So that you're aware, this idea came up a few years ago and involved a long discussion that eventually lost steam (don't remember why) [1][2][3]. There are also a couple existing PEPs [4][5] that would be applicable (my favorite is the given statement). -eric [1] https://mail.python.org/pipermail/python-ideas/2011-September/011805.html [2] https://mail.python.org/pipermail/python-ideas/2011-June/010479.html [3] https://mail.python.org/pipermail/python-ideas/2011-September/011750.html [4] "statement local namespaces" (given statement) http://www.python.org/dev/peps/pep-3150/ [5] "General purpose decorator clause" http://www.python.org/dev/peps/pep-0403/

2 2

"Read Only" namespaces
by Richard Prosser March 29, 2014

March 29, 2014

This is probably far-fetched where Python is concerned but I still think that it could be useful. It is based on an idea that I had 36 years ago! Quite simply, the principle is to allow changes to variables within a function body (for example) but not outside of it, rather like the Functional Programming paradigm. That of course excludes the use of global variables and requires functions to return whatever is needed to the outer scope. In my experience, globals are only needed to implement persistent (inter-call) states anyway, so assuming that such a feature is available in the language I believe that the "Read Only" proposal is viable. This is highly unlikely to be implemented in Python any time soon (even with an Interpreter/compiler option) and would imply some performance overhead but I still think that it is a nice idea. Any comments? Would a "Persistent Variables" PEP be interesting in its own right? Thanks ... Richard

2 1

iterating over a generator while sending values
by Clay Gerrard March 28, 2014

March 28, 2014

Hi, I often use generators when I want to bake some smarts into my producers in a typical producer and consumer pattern. Twice in the past year or so on different projects after a few iterations on my producer/consumer code ("iterations", ha!) I would eventually find I wanted my consumer to send some simple feedback to the producer - when using generators send works great for this! I normally start here: q = producer() def consume(q): for item in q: success = handle(item) But then ended up here: def consume(q): success = None while True: try: item = q.send(success) except StopIteration: break success = handle(item) But this always feels clunky; almost "out of order" - I've normally got some logging and other bits in there. I'm just trying to iterate over the queue, the StopIteration just feels like noise. It seems I always started out with a simple iteration and then end up refactoring in the send function once I realize I need to add something extra into my producer: I want to just say: def consume(q): for item in q: success = handle(item) q.send(success) # oh how I wish I could just add this here! But I can just "tack on" the send because it consumes too, so I have to rephrase the "simple" iteration into a "while True try except StopIteration break" - yuk. I normally just end up adding a little helper function, which I've named "sending" and use it like this: def consume(q): for item in sending(q, lambda: success): success = handle(item) ^ much closer to the original iterative code! Here's the implementation if you didn't already guess: # this function is missing from stdlib for some reason... def sending(g, sender): """ Iterate over g with send :params g: the iterable :params sender: the callable returning the value to send """ yield g.next() while True: yield g.send(sender()) This will raise StopIteration as soon as g is exhausted, but since the caller is already consuming "sending" with a for loop it all works ok in the end. Does anyone else have a different way of iterating over a generator while sending values? Does everyone just already have a similar function stuck in their bag? Does this happen to other folks frequently enough that a generalizable solution could be added to stdlib? Maybe it's too simple to be worth anyone's time or i'm missing something that already works better... Thanks! -clayg

4 3

Turn signal.SIG* constants into enums
by Giampaolo Rodola' March 27, 2014

March 27, 2014

In Python 3.4 some module constants were turned into enums. The affected modules were socket (AF_INET*, SOCK_*) and pslistlib (FMT_*). For 3.5 there's a plan to do the same thing for ssl's PROTOCOL_* constants: http://bugs.python.org/issue21068 To me signal module looks like another good candidate which could benefit from this. Here's a real world example: http://docs.python.org/dev/library/asyncio-eventloop.html#example-set-signa… Thoughts? -- Giampaolo - http://grodola.blogspot.com

2 2

PEP feedback loop (with tracker account)
by anatoly techtonik March 26, 2014

March 26, 2014

It is not that hard to implement an approval function for the PEP site (to improve the visibility and the process). For example, on the page http://legacy.python.org/dev/peps/pep-0231/ the field "Status: Rejected" can be linkified to bring up a popup window with short info about quantity of people who approved the PEP (content details can be extended later). It is not that hard to add a disqus-style JS form to set status of your position towards PEP using tracker account. Will that be useful for Python community? -- anatoly t.

7 12

Infix matrix-multiply, but not general infix operators?
by Jared Grubb March 25, 2014

March 25, 2014

A couple months back, Andrew Barnet brought up the idea of adding a way to apply functions in an infix style. It really didnt gain much traction. (Subject line "Infix Functions") Nathaniel Smith has just proposed a PEP to add an infix matrix multiply operator, and everyone seems to love it. I honestly am surprised at the difference in reaction. Why are we so quick to add a single-purpose punctuation mark, but reject a named infix operator that works for many general cases? Is the matrix-multiply use case so special? I dont oppose the PEP, and I dont mean to derail it. And I know there was quite a bit of discussion on Andrew's email. But I cant help feel that Andrew's idea didnt get the appreciation it should. (But hey, I'm biased because I like the idea :)) Jared QUICK SUMMARY OF ANDREW'S EMAIL: (Not a proposal for syntax or naming) m `cross` n m `dot` n a `Pair` b a `Tree` (b `Tree` c `Tree` d) `Tree` e And of course the matrix multiply (and matrix div that was briefly discussed) are implementable too: m `mmul` n m `mdiv` n

14 20

Re: [Python-ideas] Python-ideas Digest, Vol 88, Issue 199
by Colin J. Williams March 24, 2014

March 24, 2014

#4 seems to refer to the Python basic system. However numpy.array does deal with multi-dimension arrays. On 23-Mar-2014 9:08 PM, python-ideas-request(a)python.org wrote: > ------------------------------ Message: 4 Date: Mon, 24 Mar 2014 > 11:49:43 +1100 From: Steven D'Aprano <steve(a)pearwood.info> To: > python-ideas(a)python.org Subject: Re: [Python-ideas] [RFC] draft PEP: > Dedicated infix operators for matrix multiplication and matrix power > Message-ID: <20140324004943.GX16526@ando> Content-Type: text/plain; > charset=us-ascii On Sun, Mar 23, 2014 at 11:40:10AM +0100, Sturla > Molden wrote: >> On 15/03/14 01:09, Antoine Pitrou wrote: >> >>> Really? That should be up to the third-party library implementing the @ >>> operator for its types, not to the language itself: Python _suggests_ >>> an use case for @, it doesn't mandate it (especially as there's no >>> appropriate data type in the stdlib). >> array.array is an appropriate type for supporting @ for matrix >> multiplication. > No it isn't. The PEP even discusses it: > > > [quote] > array objects cannot represent multidimensional data at all, > which makes ``__matmul__`` much less useful. Second, > providing a quality implementation of matrix multiplication > is highly non-trivial. Naive nested loop implementations > are very slow and providing one in CPython would just create > a trap for users. But the alternative -- providing a modern, > competitive matrix multiply -- would require that CPython link > to a BLAS library, which brings a set of new complications. > In particular, several popular BLAS libraries (including the > one that ships by default on OS X) currently break the use of > ``multiprocessing`` [#blas-fork]_. And finally, we'd have to > add quite a bit beyond ``__matmul__`` before ``memoryview`` > or ``array.array`` would be useful for numeric work -- like > elementwise versions of the other arithmetic operators, just > to start. > [end quote] > > >

3 2

alternative-implementation friendly changes to Python Launcher for Windows
by Vernon D. Cole March 23, 2014

March 23, 2014

A new issue was recently raised on the IronPython group addressing the idea of including a copy of the *Python Launcher for Windows* in the IronPython binary distribution. http://www.python.org/dev/peps/pep-0397/ I would like to get feedback from this group, also, before opening a work item for the suggestion. Please add any comments or suggestions you may have. Here is a partial clip from our discussion... vernondcole <http://www.codeplex.com/site/users/view/vernondcole> wrote Fri > at 2:32 AM > I made a clone of the git version of PLW to look this idea over. It is a > smallish C++ program and should not be too hard to patch to make > alternate-implementation friendly. The problem today is that it is CPython > specific and searches the Windows registry to determine which versions of > Python are present. Alternate implementations are supported only using > entries in the py.ini file and setting the #! line in your source code > approprietly. (Which works great. I even have mine launching perl scripts, > just to proove that it can.) > > I propose adding two features to PLW. > > 1) extend the first command-line switch to also pick items from the py.ini > file. I would like to type: > py -ipy myprogram.py > and have the command associated with "ipy" in my py.ini file executed. > > 2) PLW should have (in py.ini) a [default] section identifying which > command in the [commands] section should be run in the absence of a > command-line switch. > > Alternate implementations should then install (or modify) py.ini to > support their interpreter. > > The extended PLW version ought to be backported to the CPython > distribution, so that any implementation of Python will work. I don't think > that the CPython group would object. > > Note 1: I have not compared to make sure the stand-alone PLW is identical > with the Python 3.3 version. > Note 2: We need to make sure that PLW will work correctly even if no > CPython versions are present. > Note 3: I fear that I have just volunteered to write the patch. Is someone > else out there willing? It's been a very long time since I wrote commercial > quality C code. > > jdhardy <http://www.codeplex.com/site/users/view/jdhardy> wrote Fri at > 5:33 AM > Sounds good. I remember from the original PLW discussion they want it to > work for other implementations but weren't sure what would be needed. It > sounds like you're making that list. :) > > It's probably worth opening bugs on the Python tracker describing what > needs to change, since it's not IronPython specific. If you don't feel > comfortable writing the code perhaps someone over there can, or at least > review the code for you. >

2 1

overloading chained comparison
by Nathaniel Smith March 19, 2014

March 19, 2014

Hi python-ideas, Since Nick is being a killjoy and shutting down bikeshedding in the @ thread [1] until the numpy folks finish bikeshedding over it themselves [2], I thought I'd throw out another idea for... brainstorming [3]. Guido has suggested that while PEP 335 (overloading 'and', 'or', 'not') is rejected, he might be amenable to making chained comparisons like a < b < c overloadable [4]. The idea is that right now, a < b < c always expands to (a < b) and (b < c) and the 'and' forces boolean coercion. When working with arrays, '<' is often used elementwise, so we want the 'and' to apply elementwise as well; in numpy, this is done using '&' (since we can't overload 'and'). So if a, b, c are numpy arrays, we'd like this to instead expand to (a < b) & (b < c) Similar considerations apply to other systems that overload the comparison operators, e.g. DSLs for generating SQL queries. This seems like a good and straightforward enough idea to me in principle, but it's not at all clear to me what the best way to accomplish it in practice is. I thought of three options, but none is obviously Right. To have a way to talk about our options precisely, let's pretend that there's something called operator.chain_comparison, and that the way it works is that a < b <= c produces a call like operator.chain_comparison([a, b, c], [operator.lt, operator.le]) Right now, the semantics are: # Current def chain_comparison(args, ops): for i, op in enumerate(ops): result = op(args[i], args[i + 1]): # short-circuit if not result: return result return result (Of course in reality in CPython the compiler unrolls the loop and inlines this directly into the bytecode, but whatever, that's an implementation detail.) IDEA 1: simple, neat and (sort of) wrong Let's define a new special method __chain_and__; whenever we do a chain comparison, we check for the presence of this method, and if found, we call it instead of using 'and'. The intuition is that if we have a < b < c then this expands to either (a < b) and (b < c) or (a < b).__chain_and__(b < c) depending on whether hasattr((a < b), "__chain_and__"). Notice that the first case is short-circuiting, and the second is not. Which seems totally fine (contra PEP 335), because short-circuiting by definition requires that you make a boolean decision (quit early/don't quit early), and the whole point of these overloads is to avoid boolean coercion. I think in general the semantics here look like: # __chain_and__ def chain_comparison(args, ops): so_far = True for i, op in enumerate(ops): result = op(args[i], args[i + 1]) if hasattr(so_far, "__chain_and__"): so_far = so_far.__chain_and__(result) else: so_far = so_far and result # short-circuit, but only if the next reduction would use 'and': if not hasattr(so_far, "__chain_and__") and not so_far: return so_far return so_far So if 'arr' is a numpy array, then code like 0 < arr < 1 will now work great, because (0 < arr) will return an array of booleans, and this array will have the __chain_and__ method, so we'll end up doing (0 < arr).__chain_and__(arr < 1) and successfully return a single array of booleans indicating which locations in 'arr' are between 0 and 1. But -- suppose that we have, say, a GUI displaying a barchart, and we have a horizontal "threshold" line across the barchart that the user control. The idea is that they can move it up and down, and all the bars that it overlaps will change color, so we can easily see which bars are above the threshold and which are below it. So given the current threshold 't' (an ordinary float or int), and a 1d array holding the various bar heights, we can request the set of overlapping bars as: 0 < x < arr Okay, first problem is that this fails because here we don't want to do (0 < x).__chain_and__(x < arr) because (0 < x) is just False. So okay, let's say we enhance the above definition to allow for __rchain_and__, and we get (x < arr).__rchain_and__(0 < x) Great! So we implement it and we test it in our program and it works -- under ordinary conditions, this spits out a nice 1d array of booleans, our code continues on its merry way, using the values in this array to decide how each bar in our plot should be colored. Until the user slides the threshold line down below zero, at which point (0 < x) starts returning False, and we short-circuit out before even evaluating (x < arr). And then our code blows up because instead of getting an array like we expected, we just get False instead. Oops. __rchain_and__ is useless. So I guess the solution is to write arr > x > 0 which will always work? I'm not sure how common this case is in practice, but I find it somewhat disturbing that with the __chain_and__ approach, a < b < c and c > b > a can return completely different values even for completely well-behaved objects, and there's nothing the person writing overload methods can do about it. Certainly the bug in our original code is not obvious to the casual reader. (And AFAICT the proposal in PEP 335 also has this problem -- in fact PEP 335 basically is the same as this proposal -- so no help there.) IDEA 2: the FULLY GENERAL solution (uh oh) So we started with a call like operator.chain_comparison([a, b, c], [operator.lt, operator.le]) Maybe what we should do is treat *this* as the basic operator, and try calling a special __chain_comparison__ method on a, b, and/or c. Of course this immediately runs into a problem, because all python's existing operators have only 1 or 2 arguments [5], not an indefinite and varying number of arguments. So we can't use the standard __X__/__rX__ dispatch strategy. We need something like multimethod dispatch. (Cue thunder, ominous music.) I know this has been discussed to death, and I haven't read the discussion, so I guess people can have fun educating me if they feel like it. But what I'd suggest in this case is to do what we did in numpy to solve a similar problem [6]. Instead of using a "real" multimethod system, just directly generalize the __X__/__rX__ trick: when looking for a candidate object to call __chain_comparison__ on, take the leftmost one that (a) hasn't been tried, and (b) doesn't have another object which is a proper subclass of its type that also hasn't been tried. If you get NotImplemented, keep trying until you run out of candidates; then fall back to the traditional 'and'-based semantics. This does solve our problematic case above: only 'arr' implements __chain_comparison__, so we have 0 < x < arr becoming arr.__chain_comparison__([0, x, arr], [operator.lt, operator.lt]) and then that can just call the underlying rich comparison operators in a non-short-circuiting loop, and combine the results using '&' instead of 'and'. But it does require a somewhat odd looking piece of machinery for the dispatch. OPTION 3: the FULLY SPECIFIC solution A nice thing about the __chain_comparison__ method is that it will actually be identical for all array-like objects, no matter which library they're defined in. So, if there are multiple array-like objects from different libraries in a single chain, it doesn't matter who gets picked to handle the overall evaluation -- any one of them is good enough, and then the actual interoperability problems are delegated to the rich comparison methods, which already have to have a strategy for dealing with them. In fact, this same __chain_comparison__ can also probably be used for just about anyone who wants to define it -- e.g., I think most DB query DSLs are going to overload & to mean "and", right? So, maybe all we need is a flag that says: if you see an object with this flag anywhere in a chain, then switch to these semantics: # &-flag def chain_comparison(args, ops): for i, op in enumerate(ops): result = op(args[i], args[i + 1]): if i == 0: combined = result else: combined &= result return combined ...and otherwise, use the standard semantics. So that's what I got. None of these approaches seems obviously Right, but they're all at least sort of viable. Anyone else got any ideas? -n [1] https://mail.python.org/pipermail/python-ideas/2014-March/027174.html [2] http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069444.html [3] "We're more of the love, bikeshedding, and rhetoric school. Well, we can do you bikeshedding and love without the rhetoric, and we can do you bikeshedding and rhetoric without the love, and we can do you all three concurrent or consecutive. But we can't give you love and rhetoric without the bikeshedding. Bikeshedding is compulsory." [4] https://mail.python.org/pipermail/python-dev/2012-March/117510.html [5] Except pow(), but that doesn't really count because it never dispatches on the third argument. [6] The next release of numpy allows non-numpy array-like classes to handle all numpy math functions in a generic way by dispatch to a special __numpy_ufunc__ method; I can provide more details on what's going on here if anyone's curious: http://docs.scipy.org/doc/numpy-dev/reference/arrays.classes.html#numpy.cla… -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

3 3

A resources tracker ?
by Tarek Ziadé March 18, 2014

March 18, 2014

Hey ResourceWarning instances that are created in some classes' __del__ function, like FileIO are a great tool to track down a program bad behavior. In network programming, that's even more important to avoid crashes, or huge leaks. But some of them are very hard to fix because we don't get much context, we just get warnings at the end of the program execution, when gc.collect is called. sys:1: ResourceWarning: unclosed file <_io.FileIO name=18 mode='wb'> sys:1: ResourceWarning: unclosed file <_io.FileIO name=17 mode='rb'> Here I just know that somewhere, 2 file descriptors where not closed. What I'd like to be able to do is to track down the origin of those warnings. Since __del__ is called asynchronously, it's impossible to track it right now (or I don't know how) What we need is a way to keep track of any resource allocation *when it happens*. Here's an idea: let's add three private functions in Python's io: def __allocate_resource(fd) => records the file descriptor that was allocated, along with the current traceback. def __free_resource(fd) => removes the fd from the list. def __is_resource_allocated(fd) => tell if the resource is in the list. These three functions, plugged in somewhere in io's classes, could be used in conjunction with ResourceWarning: when __del__ is called, if the resource was not freed - we'd be able to know where it was created. Of course these functions are just a brain dump - I have no idea how io internals work. But unless I missed it, something like I've just described is missing in Python. Cheers Tarek

3 4

Re: [Python-ideas] Please reconsider the Boolean evaluation of midnight
by Alex Rodrigues March 18, 2014

March 18, 2014

I'm +1 for this idea. It's not about whether the style is good (which its not), but its fundamentally about the fact that it will fix bugs.

5 4

Function-like Modules (better control/introspection)
by C Anthony Risinger March 17, 2014

March 17, 2014

hello, could we make modules work more like this... https://gist.github.com/xtfxme/9556806 tl;dr: __code__ object available (potentially a __self__ object), reload by calling, makes working with modules more natural/safe! ENCLOSED: modules-next.py -- C Anthony #----------------------------------------------------------------------------------------( modules-next.py ) # encoding: utf-8 # # modules-next # # C Anthony Risinger import sys m_name = 'modules_next' m_source = r"""\ if __name__ == '{0}': from . import __call__ as reexec from . import __code__ as code from . import __dict__ as ns print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) __name__ = 'modules_next_reloaded' reexec() print('>>> {{0}}: {{1}}\n{{2}}\n'.format(__name__, code, ns.keys())) else: print('!!! __name__ == {{0}}\n'.format(__name__)) def throw(): raise Exception(__name__) """.format(m_name) m_code = compile(m_source, m_name, 'exec') m_dict = { '__file__': __file__ + '.next', '__name__': m_name, '__path__': list(), '__doc__': None, } #...bind namespace/functions module = eval('lambda: None', m_dict) #...automatically expose as attributes module.__dict__ = module.__globals__ #...must be set twice, __code__ sees __dict__, but we see descriptor module.__name__ = m_name #...redefine the function body! module.__code__ = m_code #...yay! we haz legitimate module! sys.modules[m_name] = module print('--* importing...') import modules_next print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), )) print('--* importing (for real this time)...') #...import/reload by calling our func-y-module module() print('>>> {0}: {1}\n{2}\n'.format( __name__, modules_next.__code__, modules_next.__dict__.keys(), )) print('--* throwing...') modules_next.throw()

4 7

for-loop-if like list comps have?
by Christopher Welborn March 15, 2014

March 15, 2014

mylist = ['apple', 'banana', 'orange', 'grapefruit'] # Why is this okay... for item in [x for x in mylist if 'p' in x]: print(item) # But this isn't? for item in mylist if 'p' in item: SyntaxError: invalid syntax # Instead you have to do another nesting level.. for item in mylist: if 'p' in item: print(item) # Or another way. for item in mylist: if 'p' not in item: continue print(item) ...And there is 'filter', 'lambdas', and whatever else to achieve the same result. But the list comprehension is already closely related to a for-loop, and it seems natural to go from: [x for x in mylist if x] to: for x in mylist if x: Just an idea, sorry if it has been mentioned before. -- \¯\ /¯/\ \ \/¯¯\/ / / Christopher Welborn (cj) \__/\__/ / cjwelborn at live·com \__/\__/ http://welbornprod.com

3 4

[RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power
by Mark Daoust March 15, 2014

March 15, 2014

*>Aaron Meurer* asmeurer at gmail.com <python-ideas%40python.org?Subject=Re%3A%20%5BPython-ideas%5D%20Fwd%3A%20%5BRFC%5D%20draft%20PEP%3A%20Dedicated%20infix%20operators%0A%20for%20matrix%20multiplication%20and%20matrix%20power&In-Reply-To=%3C9e2519ad-0940-478c-bff0-39f47ab1e826%40googlegroups.com%3E> > *>Sat Mar 15 01:48:46 CET 2014* > > > Like Robert I have the suspicion that the best option is to make @ > > right-associative and place it just above (more tightly binding) than > > > > *. But I'll poll numpy-discussion and friends and see if anyone has > > ideas for more objective measures. > > > > I guess a common case would be a*B@x, where a is a scalar. Is it more > efficient or numerically stable to evaluate that one way or the other? I'm not a numerical linear algebra expert, but I disagree. If your matrix code needs to be optimized, there's no way around thinking about your specific situation. Optimizing matrix expressions is a tricky problem. <http://en.wikipedia.org/wiki/Matrix_chain_multiplication> http://en.wikipedia.org/wiki/Matrix_chain_multiplication http://www.youtube.com/watch?v=nVt24G_2VC0 If you want a different order use parenthesis, or build a symbolic expression. I don't think either choice of priority, or associativity will be the right often enough be worth an affront to the "principal of least surprise". Things are nice and clear the way they are. >From the docs http://docs.python.org/3.4/reference/expressions.html#binary-arithmetic-ope… : """The binary arithmetic operations have the conventional priority levels... Apart from the power operator[s], there are only two levels, one for multiplicative operators and one for additive operators""" let's keep it like this: Vote P/E/MD/AS not P/E/M/MD/AS. I think that if there is a reason to change is Greg is on the right track: *>Greg Ewing* greg.ewing at canterbury.ac.nz <python-ideas%40python.org?Subject=Re%3A%20%5BPython-ideas%5D%20%5BRFC%5D%20draft%20PEP%3A%20Dedicated%20infix%20operators%20for%0A%20matrix%20multiplication%20and%20matrix%20power&In-Reply-To=%3C53243F9D.1060802%40canterbury.ac.nz%3E> *>Sat Mar 15 12:55:09 CET 2014* >Often,matrix @ vector represents a linear operator acting on an element of a vector space. When you chain them, > A @ B @ C @ v >conceptually represents acting on v with C, then B, then A. PS: >*Nathaniel Smith* njs at pobox.com <python-ideas%40python.org?Subject=Re%3A%20%5BPython-ideas%5D%20Fwd%3A%20%5BRFC%5D%20draft%20PEP%3A%20Dedicated%20infix%20operators%0A%20for%20matrix%20multiplication%20and%20matrix%20power&In-Reply-To=%3CCAPJVwBmm%2BWDCpsRDAVOZ%3DYgRX5u7HfWZ%2BCs2NWdHev9_v8_PsQ%40mail.gmail.com%3E> >*Fri Mar 14 22:21:12 CET 2014* > >The 'matrix @@ -1' notation will probably be a nice win for >beginners and other less-sophisticated programmers ... There's a spot where I like the Facade approach: return an object that only has one method, numpy.solve as __mul__. Make it hard to do badly. >[x.M * y] This is an interesting suggestion! I think it hasn't received full consideration before ... >but this might well be the next-best thing to a real operator. Maybe It is a real operator, it's just spelled ".M*" instead of "@" ;) Thanks for all the work on this Nathaniel. Mark Daoust

1 0

Re: [Python-ideas] Fwd: [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power
by Aaron Meurer March 15, 2014

March 15, 2014

On Friday, March 14, 2014 7:46:11 PM UTC-5, Aaron Meurer wrote: On Friday, March 14, 2014 4:21:12 PM UTC-5, Nathaniel Smith wrote: > On Fri, Mar 14, 2014 at 5:53 PM, Guido van Rossum <gu...(a)python.org> > wrote: > >> > I have now read the PEP, and I think it's good. I think it's a waste of > time > >> > to keep bikeshedding on the choice of operator -- @ is the best > compromise. > >> > I do have a few specific notes: > >> > > >> > - Right associativity is not unheard of in Python. E.g. **. If you > >> > think that for other reasons @ should be right associative, don't > >> > let Python's tradition stop you. But then you need to decide which > >> > of * and @ binds more tightly -- e.g. does a*b@c mean a*(b@c) or > >> > (a*b)@c? And if you choose the latter, it follows that a@b*c means > >> > a@(b*c) -- is that okay? (And similar examples exist for the other > >> > choice.) > >> > Like Robert I have the suspicion that the best option is to make @ > >> right-associative and place it just above (more tightly binding) than > >> *. But I'll poll numpy-discussion and friends and see if anyone has > ideas for more objective measures. > I guess a common case would be a*B@x, where a is a scalar. Is it more efficient or numerically stable to evaluate that one way or the other? Aaron Meurer > > - Did you consider a duck-typing (is that the word?) attribute? > >> > E.g. a*b is elementwise multiplication; a.M*b must be used for > >> > matrix multiplication. (Your use of .T as "transpose" made me think > >> > of this.) Of course the question is, can you get those packages > >> > that currently use * for matrix multiply to comply? (I don't consider > >> > this a serious counter-proposal. But you list a bunch of rejected > >> > alternatives; this could be in that list. > >> > This is an interesting suggestion! I think it hasn't received full > >> consideration before because the tangle between numpy.ndarray and > >> numpy.matrix means that the most obvious implementation for an > >> ndarray.M attribute would be to return a full-fledged numpy.matrix > >> object... and no-one wants to encourage proliferation of numpy.matrix > >> objects. But returning a tiny special-purpose class that only > >> implements __mul__ would avoid that objection. At that point it's > >> basically a nicer version of the "user-defined infix" '*dot*' operator > >> idea. It has similar problems with needing an unaesthetically magical > >> implementation, producing a proliferation of special classes (one > >> extra one for each array type), requiring an allocation on every call, > >> etc., but this might well be the next-best thing to a real operator. > >> > All in all, I think we'd rather have a real operator, so if you're > >> happy to go with @ then I won't put lots of effort into finding > >> horrible problems that force us to reject .M ;-), but I'll certainly > >> add it to the PEP in any case. > >> > > - Is @@ really necessary? It seems you are adding it mostly because > >> > it's cute and because of the parallel with **, not because it is > >> > actually important enough to add new syntax. And then later you use > >> > it as an argument for @, which seems a bit circular. Also, if we > >> > were to make @ right-associative, the parallel with ** is already > >> > imperfect. > >> > @@ hasn't received as much attention as the rest of the proposal, so > >> I'll check in with numpy-discussion etc. on this as well. But my > >> personal feeling is +0 on @@ -- all else being equal, it's nicer to > >> have it than not. Is all else equal? Given the existence of @, the > >> increase in language complexity is small (or arguably negative, since > >> people who know *, **, @ may be surprised to find @@ missing, and have > >> to memorize its non-existence as an extra rule); the opportunity cost > >> is low (given the existence of @ I can't imagine we'll want to use the > >> token @@ for something else later); and @@ will be used in real life, > >> if not as often as @ itself -- 'vec @@ 2' for squared Euclidean length > >> probably won't be too uncommon, 'matrix @@ n' gets used for things > >> like simulating random walks on graphs or other markov chains, and the > >> 'matrix @@ -1' notation will probably be a nice win for beginners and > >> other less-sophisticated programmers who just want to get some > >> computation done in a way that doesn't require them to keep a more > >> complicated mapping between math and code notation in their head or > >> caring about numerical details. I think of @@ being like, say, '%=' -- > >> nothing one would ever add syntax for in isolation, but if designing > >> an overall system of operators then it makes more sense. > >> > Probably in the end it just comes down to your aesthetic judgement > >> :-). Numeric folk will be fine either way. > >> > (And I wouldn't consider @@ a *strong* argument for spelling '@' as > >> '@'; it's just mentioned in that section because there aren't really > >> *any* strong arguments for preferring one spelling versus another, we > >> have to make a decision, and so a weak argument is better than > >> nothing. Two useful operators for the complexity cost of one is a good > >> deal? :shrug: Obviously if @@ gets dropped I'll drop that bullet point > >> as well.) > >> > > - For better counts of usages, perhaps Sourcegraph.com might help? It > >> > is a source code query engine that has a Python parser and (limited) > >> > type inference built in (also separately available as pysonar on > >> > github IIRC). To be clear, I don't need more numbers to be convinced. > >> > Oo, that is shiny, thanks for the tip. > >> > > Once we've decided on associativity and @@, I'm ready to accept. > >> > Wonderful to hear! Thanks for giving this so much time and attention > >> (on no warning)! > >> > -n > >> > -- > >> Nathaniel J. Smith > >> Postdoctoral researcher - Informatics - University of Edinburgh > >> http://vorpus.org > >> _______________________________________________ > >> Python-ideas mailing list > >> Python...(a)python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 0

re-organizing the sys module
by Eric Snow March 14, 2014

March 14, 2014

The sys module is perhaps the most core module to Python and one of the oldest. It has grown relatively organically over the years and its contents could use a little clean up. Similar proposals have come up before but without enough interest to keep them rolling. My motivation for cleaning up sys is a combination of things: * I'm working on a proposal that may include adding a sub-namespace to sys (similar to what we did with sys.implementation) and removing-ish some attributes. * The namespace is large enough that scanning quickly through it, particularly alphabetically a la help() or the docs, isn't very helpful. * Other outstanding proposals like PEP 432 may restructure underlying APIs and having the sys module mirror the resulting APIs would help reduce the cognitive overhead in this space. * It would be nice to support descriptors on the module. * The current structure does not communicate the role of the module very well. Changing the sys module should be done carefully and as little as possible. The status quo has something like a +10 modifier when it comes to sys changes. The current API is pretty stable and undoubtedly a lot of code relies on it. This proposal reflects an understanding of those considerations and addresses them in an entirely backward compatible way (including impact on alternate Python implementations). At the moment I'm not proposing any big changes to sys nor any changes at all to its API. That can come later. <wink> Instead, this proposal is about setting the stage for better flexibility in making changes to sys when the need arises. Consequently I propose just 2.5000000000000001 relatively simple changes: 1. the existing sys module be renamed to _sys (and the stdlib be changed accordingly where appropriate); 2. a pure Python sys module be added to the stdlib that wraps the existing API; 2.5. the new module should use the replace-itself-in-sys.modules technique to provide a module subclass that supports descriptors. Thoughts? -eric

5 7

a standard approach to module-replace-itself-in-sys.modules
by Eric Snow March 14, 2014

March 14, 2014

Not that long ago we had a discussion about the esoteric, yet totally supported and not going anywhere, trick of having a module replace itself in sys.modules. Presently there isn't any convention surrounding the technique. It's simply a matter of sticking something else into sys.modules[__name__]. The problem is that there are some subtle things that can go wrong in the import system if you aren't careful and people generally won't know that. (Then again the whole technique isn't exactly in the mainstream!) To help reduce the risk here, I propose that we provide a helper in importlib.util and an associated convention. The helper will be a class decorator and the convention will be to decorate 1 class at the bottom of the module, where the class will contain any relevant customizations. The decorator will do the following: 1. create a new module subclass; 2. merge the decorated class namespace into the subclass; 3. create an instance of the subclass; 4. merge the original module namespace into the instance; 5. set sys.modules[__name__] to the new instance. For example: @importlib.util.replace_module class CustomModule: CHEDDAR = None @property def limburger (self): return None There's more to it but that captures the gist. I've been exploring API ideas, but ultimately we don't need much to get the ball rolling. In the meantime, I've published an unrefined but mostly working version of replace_module() online: https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/replace_m… -eric

3 3

numerical type combining integer and float/decimal properties [Was: Re: Python Numbers as Human Concept Decimal System]
by Wolfgang Maier March 13, 2014

March 13, 2014

Am Montag, 10. März 2014 14:53:42 UTC+1 schrieb Stefan Krah: > [My apologies for being terse, I don't have much time to follow this discussion right now.] > Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> I think users of decimal literals will just need to deal with the risk of unexpected rounding, as the alternatives are even more problematic. > That is why I think we should seriously consider moving to IEEE semantics for a decimal literal. Among other things: > - Always round the literal inputs. > - Supply IEEE contexts. > - Make Decimal64 the default. > - Add the missing rounding modes to sqrt() and exp(). > - Keep the ability to create exact Decimals through the constructor when no context is passed. > - Make the Decimal constructor use the context for rounding if it is passed. > - ... While I find this discussion about decimal literals extremely interesting, in my opinion, such a literal should have an underlying completely new numerical type, if it is really supposed to be for inexperienced users. Most of the discussions right now concern rounding issues that occur after the decimal point, but I think an at least equally big problem is rounding *to the left* of it as in (using current float): >>> 1e50 + 1000 1e+50 Importantly, Decimal is no cure here: >>> Decimal(10**50) + Decimal(100) Decimal('1.000000000000000000000000000E+50') (of course, you can debate context settings to make this particular example work, but, in general, it happens with big enough numbers.) The solution for this example is using ints of course: >>> 10**50 + 100 100000000000000000000000000000000000000000000000100 , but obviously this works only for whole numbers, so there currently is no built-in way to make this example work correctly: >>> 10**50 - 9999999999999999.5 1e+50 (or with Decimal: >>> Decimal(10**50) - Decimal('9999999999999999.5') Decimal('1.000000000000000000000000000E+50') ). If we are discussing a new number literal I would like to have it cope with this and my suggestion is a new numeric type that's sort of a hybrid between int and either Decimal or float, i.e., a type that behaves like int for digits left of the decimal point, but may truncate to the right. In pure Python, something similar (but not a literal form of course) could be implemented as a class that stores the left digits as an int and the right digits as a float/Decimal internally. Calculations involving this class would be slow due to the constant need of shifting digits between the integer´and the float part, and might be too slow for many users even when written in C, but its behavior would meet the expectation of inexperienced people better than the existing types. Going back to Mark Harris' initial proposal of unifying numeric types (which I am not trying to support in any way here), such a type would even allow to unify int and float since an ints could be considered a subset of the new type with a fractional part of zero. Cheers, Wolfgang

6 18

Python Numbers as Human Concept Decimal System
by Mark H. Harris March 12, 2014

March 12, 2014

Greetings, The purpose of this idea is to expand a bit on the decimal default idea which I submitted previously. In this idea I want to suggest the human idea of a *python number.* The concept is very simple, yet may have far reaching implications not only for future python(s) but also for the wider adaptation of python in the greater academic and professional communities. The idea of *python number* means that there are no types, no limits, no constraints, and that all *python numbers *are dynamic. The upper level syntax is also very simple; all* python numbers *are simply human. My influence for this preference is rooted in the Rexx programming language; attributed to Mike Cowlishaw, former IBM fellow. The Rexx programming language is dynamic, and has no types. To put it more succinctly for those of you who have not used Rexx, the only data type is a string of characters (that's it). *Rexx numbers* are simply those strings of characters that may be interpreted as a valid *Rexx number.* http://books.google.com/books?id=cNiVqFmPs8AC&pg=PA100&lpg=PA100&dq=rexx+nu… The Python language might be changed to adopt the *python number* concept for *all math processing*, unless explicitly modified. This goes somewhat beyond using decimal floating point as a default numerical type. It means using human numeric expressions that meet human expectation for numeric processing by default. Under the covers (whatever we mean by that) processing is handled by decimal.Decimal, unless explicitly modified. What does this mean for python users in general? Well, no more worrying about types at all... no ints, no floats, no rationals, no irrationals, no fractions, and certainly no binaries. In short, if its a human number, its a *python number.* I am expecting that (just as in Rexx numbers) defining very clearly what is a *python number* will be key for wide adaptation of the concept. But there should be no surprises for users, particularly average users. Anyone with a middle school expectation of a numeric format should be able to use *python numbers *without surprises. However, for advanced users the full interface should be available (as is the context for Decimal) through coding based on knowledge and experience, yet the default context for Decimal should be based on average users in most environments and use cases. It is my belief that Python should lead the way in the twenty-first century for advanced computation for academic, professional, business, and scientific communities. There is a 40 year momentum for embedded binary floats & doubles, also numeric types generally, but it is time to move forward. The technology is ready, and the need is great. Let's do it. {shameless plug} pdeclib https://pypi.python.org/pypi/pdeclib Thank you for your consideration. Good evening. Mark H Harris marcus

31 271

%-formatting with Decimals
by Oscar Benjamin March 12, 2014

March 12, 2014

During recent discussions I came across something unfortunate with formatting Decimals. Formatting with .format works as I would want: >>> from decimal import Decimal as D >>> '{:1.0e}'.format(D('1.123e+1000')) '1e+1000' >>> '{:1.0e}'.format(D('1.123e-1000')) '1e-1000' >>> '{:.50f}'.format(D('1.1')) '1.10000000000000000000000000000000000000000000000000' But %-formatting coerces to float: >>> '%1.0e' % D('1.123e-1000') '0e+00' >>> '%1.0e' % D('1.123e+1000') 'inf' >>> '%.50f' % D('1.1') '1.10000000000000008881784197001252323389053344726562' Am I doing this wrong? Is this just a limitation of the old-style % formatting or something that could possibly be improved? Oscar

8 13

Parsing ISO 8601 with the standard library
by Tin Tvrtković March 11, 2014

March 11, 2014

Hello, it occured to me today that it's not currently possible to parse ISO 8601 dates using the datetime.datetime.strftime function (I was parsing datetimes generated by Postgres). The problem is in the semicolon in the time zone offset. '1997-07-16T19:20:30.45+01:00' is a valid ISO 8601 date and time representation, the likes of which can be generated by the datetime module itself (the isoformat method). The %z strftime directive only recognizes offsets without the semicolon. The Python docs also direct users to inspect the platform strftime documentation; on my system the man page clearly states %z is "... The +hhmm or -hhmm numeric timezone", again, no semicolon support. Googling around, most common suggestions are to use a third party library (such as dateutil), so either this functionality really doesn't exist in a simple form in the datetime module, or is really undiscoverable. It seems to me ISO 8601 is significant enough (since there are even special methods for it in the datetime module) for it to be parsable, in a non-complex way, by the standard library. I guess it's interesting to point out a new flag was added to Java's SimpleDateFormat's date and time patterns in Java 7 - the X directive stands for "ISO 8601 time zone" (-08; -0800; -08:00, Z). I've worked with this so I happen to know it off the top of my head. Thanks in advance for comments. Tin

6 7

Suggestion for standardized annotations
by Cem Karan March 11, 2014

March 11, 2014

Hi all, I wanted to make a suggestion for a convention on how to write annotations. I know that PEP 8 says that the standard library won't use them in order to avoid setting a convention, but that could lead to confusion and conflicts in how they are used in non-standard libraries. The reason I'm thinking about this is because of how docstrings are currently used. If you use PLY (http://www.dabeaz.com/ply/) and also want to use something like Sphinx (http://sphinx-doc.org/), you're going to have a problem; PLY stores its rules in the docstring, while Sphinx parses it for documentation. I want to prevent this problem for annotations. My thought is that all annotations should be dictionaries. The keys should all be unicode strings that are uppercase UUIDs (basically, what 'uuid.uuid1().hex.upper()' returns), and the values can be anything the programmer wants. Each project can generate one (or more) UUIDs to put into the dictionary, and publicly document what the UUID's meaning is (preferably somewhere where a search engine can find it). The advantage is that since UUIDs are unique, the number of false positives you'll get while searching for it should be low; I've tested this on a different mailing list I'm on, and the UUID I generated for it has 0 false positives, while picking up the complete discussion involving it. As an example, if project A had chosen the UUID B3D2AFE8A45A11E3AE24D49A20C52EF2 and project B chose the UUID C02D7C64A45A11E39DAFD49A20C52EF2, we might annotate a function as follows: def foo(first : { B3D2AFE8A45A11E3AE24D49A20C52EF2 : {"doc" : "English documentation"}, C02D7C64A45A11E39DAFD49A20C52EF2 : {"doc" : "expression : MINUS expression %prec UMINUS"} }, second) -> {B3D2AFE8A45A11E3AE24D49A20C52EF2: {"type" : int, "doc" : "Typechecking for a linter"}}: pass You can already see the downside of this approach; it's really, really verbose. However, at least it avoids outright conflicts in usage that prevent usage of certain tools/projects together. Note that I did consider using names as keys directly (e.g. 'doc'). However, that requires a strong, universal convention on what each key means. Since we can't seem to figure that out for the docstring, I don't see why we should expect to be able to figure it out for any of proposed keys. Moreover, the set of keys would need to be documented somewhere, the documentation kept up to date, etc. It becomes a management nightmare. UUIDs have the advantage the we just tell everyone how to generate their own, and let them go at it. If someone wants to use a given project's docstring for their own purposes, it is up to them to keep the meaning the same. Thoughts/suggestions? Thanks, Cem Karan

8 13

Changing Decimal.__pos__ and Decimal.__neg__ [Was: Re: Python Numbers as Human Concept Decimal System]
by Oscar Benjamin March 10, 2014

March 10, 2014

On 10 March 2014 08:32, Mark Dickinson <dickinsm(a)gmail.com> wrote: > On Sun, Mar 9, 2014 at 8:07 PM, Oscar Benjamin <oscar.j.benjamin(a)gmail.com> > wrote: >> >> The problem though is with things like +3.14d or -3.14d. Python the >> language treats the + and - signs as not being part of the literal but >> as separate unary operators. This is unimportant for int, float and >> imaginary literals because there unary + is a no-op and unary - is >> exact. For decimal this is not the same as +Decimal('3.14') is not the >> same as Decimal('+3.14'): > > Another option would be to change the behaviour of + and - on Decimal > objects by binding `__pos__` and `__neg__` to the specification's 'copy' and > 'copy-negate' operations instead of (as currently) to the 'plus' and 'minus' > operations. Those behaviours are more in keeping with the way that + and - > operate on other objects. Hi Mark and thanks for responding, Do you think it's really feasible to make such a change? The use of unary + for rounding to context has been a feature of the decimal module for as long as I've been using it. It's shown in the docs both in the Recipes section and the FAQ. I just advised Mark Harris to do this and it know appears ~20 times in his pdeclib module: https://code.google.com/p/pythondecimallibrary/source/browse/pdeclib.py#207 I would expect that a change such as this would break quite a lot of code (some of mine would break) so there would have to be some kind of compatibility mechanism or deprecation process or something but I can't think of a good way of doing it. Oscar

7 12

Extend socket.accept to put accepted socket in non-blocking mode
by Saúl Ibarra Corretgé March 9, 2014

March 9, 2014

Hi, (I hope this list is the right place for this) I was browsing around for accept4 usage in Python and I saw it's already used internally in socket.accept (on Linux only), albeit it's not exposed, so accepting an incoming connection and setting it to be non-blocking takes 3 syscalls. accept4 allows us to do this with a single syscall, and it made it into FreeBSD 10, so that's another system that could benefit from this optimization. Would it be desirable to extend socket.accept to something like: socket.accept(set_non_blocking=False) which would use accept4's flags on supported systems and fallback to regular fcntl on others? The idea is to accept an incoming connection and make it non-blocking at the same time, asyncio and other frameworks would benefit from this by doing one function call and 2 syscalls less (on supported systems, that is). Cheers, -- Saúl Ibarra Corretgé bettercallsaghul.com

4 5

Enhance exceptions by attaching some more information to them
by Sebastian Kreft March 8, 2014

March 8, 2014

More than once I've been in a situation where I wish that some of the stdlib exceptions had a better message or some more information to help me diagnose the problem. For example: a = [1, 2, 3, 4, 5] a[5] IndexError: list index out of range In this case there's no reference to neither the length of the array nor to the offending index. I'm of the idea that we could extend the exceptions by adding some more information to them, so 3rd party libraries could use them for debugging/logging. For example, one such use case would be to have a modified test runner, that in case of exceptions automatically prints some debug information. Another would be a logger system that in case of an exception also logs some debug info that could be relevant to understand and solve the issue. I propose extending (at least) the following exceptions with the following attributes: KeyError: key, object IndexError: index, object AttributeError: attribute, object NameError: name Of course that populating these attributes could be controlled by a flag. I know that some of this information is already present in some exceptions, depending on the context. However, I propose adding these attributes, as in this way a tool could easily and reliably extract the information and work with it, as opposed to have to parse the huge variety of different messages there are. For the first use case mentioned above I have a working prototype, although it does not use this proposal, but a series of hacks (I'm modifying the bytecode to save a reference to the key and object :() and parsing of the exception messages. But I want to share what the output of such a tool could be. ====================================================================== ERROR: test_attribute (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 18, in test_attribute AttributeError: 'str' object has no attribute 'Lower'. Did you mean 'islower', 'lower'? Debug info: Object: '' Type: <type 'str'> ====================================================================== ERROR: test_index (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 6, in test_index IndexError: list index out of range Debug info: Object: [1, 2] Object len: 2 Index: 2 ====================================================================== ERROR: test_key (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 10, in test_key KeyError_: 'fooo', did you mean 'foo'? Debug info: Object: {'foo': 1} Key: 'fooo' ====================================================================== ERROR: test_name (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 14, in test_name NameError: global name 'fooo' is not defined. Did you mean 'foo'? ---------------------------------------------------------------------- Ran 4 tests in 0.005s -- Sebastian Kreft

4 4

Re: [Python-ideas] Python Numbers as Human Concept Decimal System
by Guido van Rossum March 8, 2014

March 8, 2014

[CC back to the list because you posted the same argument there but without the numerical example, and my working through that might help others understand your point] On Fri, Mar 7, 2014 at 9:18 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > The main point I'm getting at is that by rounding 0.100000000000000012 to > 0.1 instead of 0.10000000000000000555..., You're no longer rounding it to > the nearest binary float, but instead to the second nearest > Decimal(repr(binary float)) (since 0.10000000000000002 is closer than > 0.1). > OK, let me walk through that carefully. Let's name the exact mathematical values and assign them to strings: >>> a = '0.100000000000000012' >>> b = '0.1000000000000000055511151231257827021181583404541015625' >>> c = '0.10000000000000002' Today, Decimal(float(a)) == Decimal(b). Under my proposal, Decimal(float(a)) == Decimal('0.1'). The difference between float('0.1') and float(c) is 1 ulp (2**-56), and a is between those, but closer to c; but it is even closer to b (in the other direction). IOW for the mathematical values, 0.1 < b < a < c, where a is closer to b than to c, So if the choices for rounding a would be b or c, b is preferred. So far so good. (And still good if we replace c with the slightly smaller exact value of float(c).) And your point is that if we change the allowable choices to '0.1' or c, we find that float(b) == float('0.1'), but a is closer to c than to 0.1. This is less than 1 ulp, but more than 0.5 ulp. I find the argument intriguing, but I blame it more on what happens in float(a) than in what Decimal() does to the resulting value. If you actually had the string a, and wanted to convert it to Decimal, you would obviously write Decimal(a), not Decimal(float(a)), so this is really only a problem when someone uses a as a literal in a program that is passed to Decimal, i.e. Decimal(0.100000000000000012). That's slightly unfortunate, but easy to fix by adding quotes. The only place where I think something like this might occur in real life is when someone copies a numerical recipe involving some very precise constants, and mindlessly applies Decimal() without string quotes to the constants. But that's a "recipe" for failure anyway, since if the recipe really uses more precision than IEEE double can handle, *with* the quotes the recipe would be calculated more exactly anyway. Perhaps another scenario would be if the constant was calculated (by the recipe-maker) within 0.5 ulp using IEEE double and rendered with exactly the right number of digits. But these scenarios sound like either they should use the quotes anyway, or the calculation would be better off done in double rather than Decimal. So I think it's still pretty much a phantom problem. > Of course that's not true for all reals (0.1 being the obvious > counterexample), but it's true for some with your proposal, while today > it's true for none. So the mean absolute error in Decimal(repr(f)) across > any range of reals is inherently higher than Decimal.from_float(f). Put > another way, you're adding additional rounding error. That additional > rounding error is still less than the rule-of-thumb cutoff that people use > when talking about going through float, but it's nonzero and not guaranteed > to cancel out. > > On top of that, the distribution of binary floats is uniform (well, more > complicated than uniform because they have an exponent as well as a > mantissa, but you know what I mean); the distribution of closest-repr > values to binary floats is not. > > I have no idea whether either of these are properties that users of > Decimal (or, rather, Decimal and float together) care about. But they are > properties that Decimal(float) has today that would be lost. > -- --Guido van Rossum (python.org/~guido)

2 1

Please reconsider the Boolean evaluation of midnight
by Shai Berger March 8, 2014

March 8, 2014

Hi all, This is my first post here, following a recommendation from Alexander Belopolsky to use this list, to try to convince the Python developers to reopen a ticket. I am a long-time Python user, and a Django committer. http://bugs.python.org/issue13936 is a complaint about the fact that midnight -- datetime.time(0,0,0) -- is evaluated as False in Boolean contexts. It was closed as invalid, under the claim that """It is odd, but really no odder than "zero values" of other types evaluating to false in Boolean contexts""". I would like to ask for this to be reconsidered; since the ticket was closed, two main arguments were given for this: 1) The practical argument (paraphrasing Danilo Bergen and Andreas Pelme): The current semantics is surprising and useless; users do not expect valid times to be falsey, and do not normally want code to have special behavior on midnight. Users who ask for Boolean evaluation of a variable that's supposed to hold a time value, usually write "if var" as shorthand for "if var is not None". 2) The principled argument (which I think is at the root of the practical argument); quoting myself from the ticket: """Midnight is not a "zero value", it is just a value. It does not have any special qualities analogous to those of 0, "", or the empty set. ... Midnight evaluting to false makes as much sense as date(1,1,1) -- the minimal valid date value -- evaluating to false""". Thanks, Shai.

37 219

Add "goto fail" to Python?
by Sturla Molden March 7, 2014

March 7, 2014

"goto fail" is a well-known error handling mechanism in open source software, widely reputed for its robusteness: http://opensource.apple.com/source/Security/Security-55471/libsecurity_ssl/… https://www.gitorious.org/gnutls/gnutls/source/6aa26f78150ccbdf0aec1878a41c… I believe Python needs to add support for this superior paradigm. It would involve a new keyword "fail" and some means of goto'ing to it. I suggest "raise to fail": if (some_error): raise to fail fail: <error handling code> Unless there are many objections, this fantastic idea might be submitted in a (short) PEP somewhere around the beginning of next month. There is some obvious overlap with the rejected "goto PEP" (PEP 3163) and the Python 2.3 goto module. However, the superiority of goto fail as error generation and error handling paradigm has since then been thoroughly proven. http://legacy.python.org/dev/peps/pep-3136/ http://entrian.com/goto/download.html Regards, Sturla Molden

9 10

Re: [Python-ideas] Another way to avoid clumsy lambdas, while adding new functionality
by Carl Smith March 6, 2014

March 6, 2014

Just for the record, I never called them bills just because they contain a dollar sign! I was thinking of a short word which describes an expression who's evaluation changes depending on the local circumstances. Like a legal bill, not a dollar bill. The name macro doesn't really work, as it's only an expression, it's not a code block. --- The idea was never to start passing complex expressions around the place. It was just to allow a function to take an expression containing about one or two names, and evaluate the expression internally, so this... func(lambda foo: foo > 0) ...can become... func($foo > 0) Other uses are there, and might be helpful from time to time, but it's mainly about cleaning up APIs. I personally work with user facing Python APIs all day, and users are generally programming interactively, so perhaps my take is atypical. --- What the hell is a thunk anyway? It's a horrible name. --- On the jQuery like syntax, $(foo), that might work and might do away with the issue of expressions containing bills always evaluating to bills. I took that too far. Good point. Cheers

6 7

One more time... lambda function <--- from *** signature def.
by Ron Adam March 6, 2014

March 6, 2014

Starting new thread because this bike has a different shape and color. Yesterday I was thinking that just making the keyword lambda assignable like True, False, and None, would be enough. But the issue with that is lambda isn't a name to an actual object or type. That was the seed for this idea. How to get lambda like functionality into some sort of object that would be easy to use and explain. This morning I thought we could have in a functions definition something, like "*", and "**", to take an expression. Similar to Nicks idea with =:, but more general. The idea is to have "***" used in def mean to take "any" call expression and not evaluate it until *** is used on it. ie... the same rules as *. When used in a def to pack a tuple, and when used outside def, to unpack it. So, "***" used in a def, stores the call expression, at call time, and when used later, expresses it. A function call that captures an expression may be tricky to do. Here's one approach that requires sugar when a function defined with "***" is called. class TriStar: def __init__(self, expr): """ expr is a callable that takes no arguments. """ self.expr = expr def __tristar__(self): """ ***obj --> result """ return self.expr() def fn(***expr):... (Any other suggestions for how to do this would be good.) And at call time.... fn(...) --> fn(TriStar(expr=lambda:...)) So presuming we can do something like the above, the first case is ... def star_fn(***expr) return ***expr ... = star_fn(...) Which is a function that just returns whatever it's input is, and is even more general than using *args, **kwds. The call signature stored in expr isn't evaluated until it's returned with ***expr. So the evaluation is delayed, or lazy, but it's still explicit and very easy to read. This returns a lambda-like function. def star_lambda(***expr): return expr And is used this way... result = star_lambda(a * b + c) # captures expression. actual_result = ***result # *** resolves "result" here! The resolution is done with ***name, rather than name(). That's actually very good because it can pass through callable tests. So you can safely pass callable objects around without them getting called at the wrong time or place. We can shorten the name because star_lambda is just a function. L = star_lambda To me this is an exceptionally clean solution. Easy to use, and not to hard to explain. Seems a lot more like a python solution to me as well. Hoping it doesn't get shot down too quickly, Ron ;-)

11 40

Another way to avoid clumsy lambdas, while adding new functionality
by Carl Smith March 5, 2014

March 5, 2014

The tentatively proposed idea here is using dollar signed expressions to define 'bills'. A bill object is essentially an expression which can be evaluated any number of times, potentially in different scopes. The following expression [a bill literal] would be pointless, but would define a bill that always evaluates to 1. $1 So, eval($1)==1. Some better examples... * assign a bill to `a` so that `a` will evaluate to the value of the name `foo` any time that `a` is evaluated, in the scope of that evaluation a = $foo * as above, but always plus one a = $foo + 1 * make `a` a bill that evaluates to the value of the name `foo` at the time that `a` is evaluated, in that scope, plus the value of `bar` **at the time and in the scope of the assignment to `a`** a = $foo + bar Note. Similarly to mixing floats with ints, any expression that contains a bill evaluates to a bill, so if `a` is a bill, `b=a+1` makes `b` a bill too. Passing a bill to eval should be the obvious way to get the value. The point? It allows functions to accept bills to use internally. The function would specify any names the bill can reference in the function's API, like keywords. def f(b): # the bill arg `b` can reference `item` for item in something: if eval(b): return True f($item < 0) You could also use a function call, for example `$foo()` would evaluate to a bill that evaluates to a call to `foo` in the scope and at the time of any evaluation of the bill. I've no idea if this is even possible in Python, and have no hope of implementing it, but thought I'd share :)

10 20

Python3.3 Decimal Library Released
by Mark H. Harris March 5, 2014

March 5, 2014

Greetings, Python3.3 Decimal Library v0.3 is Released here: https://code.google.com/p/pythondecimallibrary/ pdeclib.py is the decimal library, pilib.py is the PI library. pdeclib.py provides scientific and transcendental functions for the C Accelerated Decimal module written by Stefan Krah. The library is open source, GLPv3, comprised of two py files. My idea for python is to two things really, 1) make floating point decimal the default floating point type in python4.x, and 2) make these functions ( pdeclib.py ) or equiv available in python4.x by default. Thank you for your consideration. -- Kind regards, Mark H. Harris

14 29

Re: [Python-ideas] PEP-257: drop recommendation for extra line at end of multi-line docstring
by Guido van Rossum March 4, 2014

March 4, 2014

Removed. On Tue, Mar 4, 2014 at 8:51 AM, Simon Kennedy <sffjunkie(a)gmail.com> wrote: > On Monday, 3 March 2014 18:55:17 UTC, Ziad Sawalha wrote: >> >> Thanks, Guido. >> >> I'll follow up with updates to common tools as I come across them (ex. >> pep257: https://github.com/GreenSteam/pep257/pull/64). >> >> Ziad >> >> > The footnote's still in the PEP text. > > --Simon > -- --Guido van Rossum (python.org/~guido)

1 0

Local scope for statement blocks
by Brian Nguyen March 4, 2014

March 4, 2014

do: nonlocal x k = init for i in list: k = f(i,k) n = f2(n,k)

3 2

PEP-257: drop recommendation for extra line at end of multi-line docstring
by Ziad Sawalha March 3, 2014

March 3, 2014

PEP-257 includes this recommendation: “The BDFL [3] recommends inserting a blank line between the last paragraph in a multi-line docstring and its closing quotes, placing the closing quotes on a line by themselves. This way, Emacs' fill-paragraph command can be used on it.” I believe emacs no longer has this limitation. "If you do fill-paragraph in emacs in Python mode within a docstring, emacs already ignores the closing triple-quote. In fact, the most recent version of emacs supports several different docstring formatting styles and gives you the ability to switch between them.” - quoting Kevin L. Mitchell who is more familiar with emacs than I am. I’m considering removing that recommendation and updating some of the examples in PEP-257, but I’d like some thoughts from this group before I submit the patch. Any thoughts or references to conversations that may have already been had on this topic? Regards, Ziad

6 13

A python bridge between versions
by ian o March 2, 2014

March 2, 2014

The most quoted reason for staying with python 2 is that some required library is not available to support python 3. All it takes is one legacy library module in python 2 to keep an entire project in python 2. Many significant projects (e.g web2py) continue in python 2 for this reason. Even projects that produce code that can work under either python 2 or python 3, are themselves trapped into only using python 2 features. This is really holding back python! You cannot just import python2 code into the python 3 interpreter as it is not compatible. But could this be solved by a new approach which is to treat python 2 as another language and not directly import the code but handle python 2 code with an an extension module, which acts as bridge between versions. I mean, python three program can access C/C++ code (which is not python3), why can't the access python 2 code by treating it like C/C++? This would require an extension C module called from python 3. How to do this is well known. This extension C module would itself call the required module library(s) from python2. Calling python from C is also a well known technique. Ptyhon can call C. C can call python. So python3 call C which calls python2. So in python3 imput <python2-module> # this usually won't work because of language changes becomes import python2 <python2-module> = python2.import('<python2-module>') The resultant imported module would appear as a c++ extension to python3 and in effect be a wrapper for the python2 module using the python2 extension to execute the code This would mean programs using python2 imports have the overhead of both interpreters running in memory and calls across the python3/python2 boundary are 'wrapped' as c++ calls that then activate the required code within python 2. In reality quite a small overhead on modern computers. python2.sys.path (within the 'python2' module) or similar would be needed for the separate import space running inside the python2 boundary. Callbacks back across the python3/python2 boundary back into python3 are a further complication but several possible solutions exist to deliver this. Not many library modules require callback functions to the library module itself so this not the central issue. There is some work for the person coding the python 3 app in terms of a different syntax for legacy module imports and potentially setting python 2 environment values (like python2.sys.path), but not a lot and the python3 is the new program being worked on as opposed to the legacy library which can remain untouched. I think this type of approach could change the recommendation on 'python 2 or python 3' to a simple 'get the latest available to you and here is how to use legacy libraries if you have the need'. Thoughts or criticisms?

14 29

Re: [Python-ideas] Update the required C compiler for Windows to a supported version.
by Steve Dower March 2, 2014

March 2, 2014

Stephen J. Turnbull wrote: > Vernon D. Cole writes: > >> I cannot compile a Python extension module with any Microsoft compiler >> I can obtain. > > Your pain is understood, but it's not simple to address it. FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. > The basic problem is that the ABI changes. Therefore it's going to require > a complete new set of *all* C extensions for Windows, and the duplication > of download links for all those extensions from quite a few different vendors > is likely to confuse a lot of users. Specifically, the CRT changes. The CRT is an interesting mess of data structures that are exposed in header files, which means while you can have multiple CRTs loaded, they cannot touch each other's data structures at all or things will go bad/crash, and there's no nice way to set it up to avoid this (my colleague who currently owns MSVCRT suggested a not-very-nice way to do it, but I don't think it's going to be reliable enough). Python's stable ABI helps, but does not solve this problem. The file APIs are the worst culprits. The layout of FILE* objects can and do change between CRT versions, and file descriptors are simply indices into an array of these objects that is exposed through macros rather than function calls. As a result, you cannot mix either FILE pointers or file descriptors between CRTs. The only safe option is to build with the matching CRT, and for MSVCRT, this means with the matching compiler. It's unfortunate, and the responsible teams are well aware of the limitation, but it's history at this point, so we have no choice but to work with it. Cheers, Steve

4 3

Re: [Python-ideas] Update the required C compiler for Windows to a supported version.
by Steve Dower March 1, 2014

March 1, 2014

Not at my desk, so I can only get you started (didn't want to keep you waiting until Monday). It'll look something like this: IF %1 eq x86 CALL bin\vcvars32.bat %* IF %1 eq amd64 CALL bin\vcvars64.bat %* (similar for x86_amd64) The file should go at C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat If you want to redistribute something that will work for others, it's possible to monkey patch distutils from your setup.py. I can't remember the specifics right now, so I can't help with that until I'm back at work. I think that's the best long term solution for 2.7 packages that want to build on Windows. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Vernon D. Cole<mailto:vernondcole@gmail.com> Sent: ‎3/‎1/‎2014 2:24 To: Steve Dower<mailto:Steve.Dower@microsoft.com> Cc: Stephen J. Turnbull<mailto:stephen@xemacs.org>; python-ideas<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] Update the required C compiler for Windows to a supported version. Sorry to resurrect this thread after so long a time, but I am starting to work on a project where I need this again... Any chance that I can get a copy (or a link to) " the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one" ? It's only simple to add one if you know what changes to make. -- VC On Wed, Nov 27, 2013 at 7:24 PM, Steve Dower <Steve.Dower(a)microsoft.com<mailto:Steve.Dower@microsoft.com>> wrote: Stephen J. Turnbull wrote: > Vernon D. Cole writes: > >> I cannot compile a Python extension module with any Microsoft compiler >> I can obtain. > > Your pain is understood, but it's not simple to address it. FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. > The basic problem is that the ABI changes. Therefore it's going to require > a complete new set of *all* C extensions for Windows, and the duplication > of download links for all those extensions from quite a few different vendors > is likely to confuse a lot of users. Specifically, the CRT changes. The CRT is an interesting mess of data structures that are exposed in header files, which means while you can have multiple CRTs loaded, they cannot touch each other's data structures at all or things will go bad/crash, and there's no nice way to set it up to avoid this (my colleague who currently owns MSVCRT suggested a not-very-nice way to do it, but I don't think it's going to be reliable enough). Python's stable ABI helps, but does not solve this problem. The file APIs are the worst culprits. The layout of FILE* objects can and do change between CRT versions, and file descriptors are simply indices into an array of these objects that is exposed through macros rather than function calls. As a result, you cannot mix either FILE pointers or file descriptors between CRTs. The only safe option is to build with the matching CRT, and for MSVCRT, this means with the matching compiler. It's unfortunate, and the responsible teams are well aware of the limitation, but it's history at this point, so we have no choice but to work with it. Cheers, Steve

1 0