Mailman 3 January 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 17 Dec '20

17 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson 18 Jan '17

18 Jan '17

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

5 13

solving multi-core Python
by Eric Snow 04 Jun '16

04 Jun '16

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort.

26 130

Prevent importing yourself?
by Ned Batchelder 02 Mar '16

02 Mar '16

Hi, A common question we get in the #python IRC channel is, "I tried importing a module, but I get an AttributeError trying to use the things it said it provided." Turns out the beginner named their own file the same as the module they were trying to use. That is, they want to try (for example) the "azure" package. So they make a file called azure.py, and start with "import azure". The import succeeds, but it has none of the contents the documentation claims, because they have imported themselves. It's baffling, because they have used the exact statements shown in the examples, but it doesn't work. Could we make this a more obvious failure? Is there ever a valid reason for a file to import itself? Is this situation detectable in the import machinery? --Ned.

14 24

Explicit variable capture list
by haael＠interia.pl 18 Feb '16

18 Feb '16

Hi C++ has a nice feature of explicit variable capture list for lambdas: int a = 1, b = 2, c = 3; auto fun = [a, b, c](int x, int y){ return a + b + c + x + y}; This allows easy construction of closures. In Python to achieve that, you need to say: def make_closure(a, b, c): def fun(x, y): return a + b + c + x + y return def a = 1 b = 2 c = 3 fun = make_closure(a, b, c) My proposal: create a special variable qualifier (like global and nonlocal) to automatically capture variables a = 1 b = 2 c = 3 def fun(x, y): capture a, b, c return a + b + c + x + y This will have an effect that symbols a, b and c in the body of the function have values as they had at the moment of function creation. The variables a, b, c must be defined at the time of function creation. If they are not, an error is thrown. The 'capture' qualifier may be combined with keywords global and nonlocal to change lookup behaviour. To make it more useful, we also need some syntax for inline lambdas. I.e.: a = 1 b = 2 c = 3 fun = lambda[a, b, c] x, y: a + b + c + x + y Thanks, haael

28 96

OrderedCounter and OrderedDefaultDict
by Ian Foote 13 Feb '16

13 Feb '16

I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: from collections import Counter, OrderedDict class OrderedCounter(Counter, OrderedDict): 'Counter that remembers the order elements are first seen' def __repr__(self): return '%s(%r)' % (self.__class__.__name__, OrderedDict(self)) def __reduce__(self): return self.__class__, (OrderedDict(self),) from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict… which suggests: |fromcollections importOrderedDict,CallableclassDefaultOrderedDict(OrderedDict):# Source: http://stackoverflow.com/a/6190500/562769def__init__(self,default_factory=N… isnotNoneandnotisinstance(default_factory,Callable)):raiseTypeError('first argument must be callable')OrderedDict.__init__(self,*a,**kw)self.default_factory =default_factory def__getitem__(self,key):try:returnOrderedDict.__getitem__(self,key)exceptKeyError:returnself.__missing__(key)def__missing__(self,key):ifself.default_factory isNone:raiseKeyError(key)self[key]=value =self.default_factory()returnvalue def__reduce__(self):ifself.default_factory isNone:args =tuple()else:args =self.default_factory,returntype(self),args,None,None,self.items()defcopy(self):returnself.__copy__()def__copy__(self):returntype(self)(self.default_factory,self)def__deepcopy__(self,memo):importcopy returntype(self)(self.default_factory,copy.deepcopy(self.items()))def__repr__(self):return'OrderedDefaultDict(%s, %s)'%(self.default_factory,OrderedDict.__repr__(self))| This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. Thanks, Ian F

14 23

A bit meta
by Michael Selik 05 Feb '16

05 Feb '16

One defect of a mailing list is the difficulty of viewing a weighted average of opinions. The benefit is that anyone can voice an opinion. This is more like the Senate than the House -- Rhode Island appears (on paper) to have as much influence as California. Luckily, we have a form of President. I'm guessing a House occurs in a more private mode of communication? Perhaps as the community gets larger, a system like StackOverflow might be a better tool for handling things like Python-Ideas. > On Jan 27, 2016, at 12:58 PM, Sjoerd Job Postmus <sjoerdjob(a)sjec.nl> wrote: > (not sure if I even have the right to vote here, given that I'm not a > core developer, but just giving my opinion)

28 100

PEP 511: API for code transformers
by Victor Stinner 31 Jan '16

31 Jan '16

Hi, This PEP 511 is part of a serie of 3 PEP (509, 510, 511) adding an API to implement a static Python optimizer specializing functions with guards. If the PEP is accepted, it will solve a long list of issues, some issues are old, like #1346238 which is 11 years old ;-) I found 12 issues: * http://bugs.python.org/issue1346238 * http://bugs.python.org/issue2181 * http://bugs.python.org/issue2499 * http://bugs.python.org/issue2506 * http://bugs.python.org/issue4264 * http://bugs.python.org/issue7682 * http://bugs.python.org/issue10399 * http://bugs.python.org/issue11549 * http://bugs.python.org/issue17068 * http://bugs.python.org/issue17430 * http://bugs.python.org/issue17515 * http://bugs.python.org/issue26107 I worked to make the PEP more generic that "this hook is written for FAT Python". Please read the full PEP to see a long list of existing usages in Python of code transformers. You may read again the discussion which occurred 4 years ago about the same topic: https://mail.python.org/pipermail/python-dev/2012-August/121286.html (the thread starts with an idea of AST optimizer, but is moves quickly to a generic API to transform the code) Thanks to Red Hat for giving me time to experiment on this. Victor HTML version: https://www.python.org/dev/peps/pep-0510/#changes PEP: 511 Title: API for code transformers Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Propose an API to register bytecode and AST transformers. Add also ``-o OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o noopt`` disables the peephole optimizer. Raise an ``ImportError`` exception on import if the ``.pyc`` file is missing and the code transformers required to transform the code are missing. code transformers are not needed code transformed ahead of time (loaded from ``.pyc`` files). Rationale ========= Python does not provide a standard way to transform the code. Projects transforming the code use various hooks. The MacroPy project uses an import hook: it adds its own module finder in ``sys.meta_path`` to hook its AST transformer. Another option is to monkey-patch the builtin ``compile()`` function. There are even more options to hook a code transformer. Python 3.4 added a ``compile_source()`` method to ``importlib.abc.SourceLoader``. But code transformation is wider than just importing modules, see described use cases below. Writing an optimizer or a preprocessor is out of the scope of this PEP. Usage 1: AST optimizer ---------------------- Transforming an Abstract Syntax Tree (AST) is a convenient way to implement an optimizer. It's easier to work on the AST than working on the bytecode, AST contains more information and is more high level. Since the optimization can done ahead of time, complex but slow optimizations can be implemented. Example of optimizations which can be implemented with an AST optimizer: * `Copy propagation <https://en.wikipedia.org/wiki/Copy_propagation>`_: replace ``x=1; y=x`` with ``x=1; y=1`` * `Constant folding <https://en.wikipedia.org/wiki/Constant_folding>`_: replace ``1+1`` with ``2`` * `Dead code elimination <https://en.wikipedia.org/wiki/Dead_code_elimination>`_ Using guards (see the `PEP 510 <https://www.python.org/dev/peps/pep-0510/>`_), it is possible to implement a much wider choice of optimizations. Examples: * Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used as iterable * `Loop unrolling <https://en.wikipedia.org/wiki/Loop_unrolling>`_ * Call pure builtins: replace ``len("abc")`` with ``3`` * Copy used builtin symbols to constants * See also `optimizations implemented in fatoptimizer <https://fatoptimizer.readthedocs.org/en/latest/optimizations.html>`_, a static optimizer for Python 3.6. The following issues can be implemented with an AST optimizer: * `Issue #1346238 <https://bugs.python.org/issue1346238>`_: A constant folding optimization pass for the AST * `Issue #2181 <http://bugs.python.org/issue2181>`_: optimize out local variables at end of function * `Issue #2499 <http://bugs.python.org/issue2499>`_: Fold unary + and not on constants * `Issue #4264 <http://bugs.python.org/issue4264>`_: Patch: optimize code to use LIST_APPEND instead of calling list.append * `Issue #7682 <http://bugs.python.org/issue7682>`_: Optimisation of if with constant expression * `Issue #10399 <https://bugs.python.org/issue10399>`_: AST Optimization: inlining of function calls * `Issue #11549 <http://bugs.python.org/issue11549>`_: Build-out an AST optimizer, moving some functionality out of the peephole optimizer * `Issue #17068 <http://bugs.python.org/issue17068>`_: peephole optimization for constant strings * `Issue #17430 <http://bugs.python.org/issue17430>`_: missed peephole optimization Usage 2: Preprocessor --------------------- A preprocessor can be easily implemented with an AST transformer. A preprocessor has various and different usages. Some examples: * Remove debug code like assertions and logs to make the code faster to run it for production. * `Tail-call Optimization <https://en.wikipedia.org/wiki/Tail_call>`_ * Add profiling code * `Lazy evaluation <https://en.wikipedia.org/wiki/Lazy_evaluation>`_: see `lazy_python <https://github.com/llllllllll/lazy_python>`_ (bytecode transformer) and `lazy macro of MacroPy <https://github.com/lihaoyi/macropy#lazy>`_ (AST transformer) * Change dictionary literals into collection.OrderedDict instances * Declare constants: see `@asconstants of codetransformer <https://pypi.python.org/pypi/codetransformer>`_ * Domain Specific Language (DSL) like SQL queries. The Python language itself doesn't need to be modified. Previous attempts to implement DSL for SQL like `PEP 335 - Overloadable Boolean Operators <https://www.python.org/dev/peps/pep-0335/>`_ was rejected. * Pattern Matching of functional languages * String Interpolation, but `PEP 498 -- Literal String Interpolation <https://www.python.org/dev/peps/pep-0498/>`_ was merged into Python 3.6. `MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of examples and use cases. This PEP does not add any new code transformer. Using a code transformer will require an external module and to register it manually. See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python obfuscator, deobfuscator, and user-assisted decompiler. Usage 3: Disable all optimization --------------------------------- Ned Batchelder asked to add an option to disable the peephole optimizer because it makes code coverage more difficult to implement. See the discussion on the python-ideas mailing list: `Disable all peephole optimizations <https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_. This PEP adds a new ``-o noopt`` command line option to disable the peephole optimizer. In Python, it's as easy as:: sys.set_code_transformers([]) It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add mechanism to disable optimizations. Usage 4: Write new bytecode optimizers in Python ------------------------------------------------ Python 3.6 optimizes the code using a peephole optimizer. By definition, a peephole optimizer has a narrow view of the code and so can only implement basic optimizations. The optimizer rewrites the bytecode. It is difficult to enhance it, because it written in C. With this PEP, it becomes possible to implement a new bytecode optimizer in pure Python and experiment new optimizations. Some optimizations are easier to implement on the AST like constant folding, but optimizations on the bytecode are still useful. For example, when the AST is compiled to bytecode, useless jumps can be emited because the compiler is naive and does not try to optimize anything. Use Cases ========= This section give examples of use cases explaining when and how code transformers will be used. Interactive interpreter ----------------------- It will be possible to use code transformers with the interactive interpreter which is popular in Python and commonly used to demonstrate Python. The code is transformed at runtime and so the interpreter can be slower when expensive code transformers are used. Build a transformed package --------------------------- It will be possible to build a package of the transformed code. A transformer can have a configuration. The configuration is not stored in the package. All ``.pyc`` files of the package must be transformed with the same code transformers and the same transformers configuration. It is possible to build different ``.pyc`` files using different optimizer tags. Example: ``fat`` for the default configuration and ``fat_inline`` for a different configuration with function inlining enabled. A package can contain ``.pyc`` files with different optimizer tags. Install a package containing transformed .pyc files --------------------------------------------------- It will be possible to install a package which contains transformed ``.pyc`` files. All ``.pyc`` files with any optimizer tag contained in the package are installed, not only for the current optimizer tag. Build .pyc files when installing a package ------------------------------------------ If a package does not contain any ``.pyc`` files of the current optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are created during the installation. Code transformers of the optimizer tag are required. Otherwise, the installation fails with an error. Execute transformed code ------------------------ It will be possible to execute transformed code. Raise an ``ImportError`` exception on import if the ``.pyc`` file of the current optimizer tag is missing and the code transformers required to transform the code are missing. The interesting point here is that code transformers are not needed to execute the transformed code if all required ``.pyc`` files are already available. Code transformer API ==================== A code transformer is a class with ``ast_transformer()`` and/or ``code_transformer()`` methods (API described below) and a ``name`` attribute. For efficiency, do not define a ``code_transformer()`` or ``ast_transformer()`` method if it does nothing. The ``name`` attribute (``str``) must be a short string used to identify an optimizer. It is used to build a ``.pyc`` filename. The name must not contain dots (``'.'``), dashes (``'-'``) or directory separators: dots are used to separated fields in a ``.pyc`` filename and dashes areused to join code transformer names to build the optimizer tag. .. note:: It would be nice to pass the fully qualified name of a module in the *context* when an AST transformer is used to transform a module on import, but it looks like the information is not available in ``PyParser_ASTFromStringObject()``. code_transformer() ------------------ Prototype:: def code_transformer(code, consts, names, lnotab, context): ... return (code, consts, names, lnotab) Parameters: * *code*: the bytecode (``bytes``) * *consts*: a sequence of constants * *names*: tuple of variable names * *lnotab*: table mapping instruction offsets to line numbers (``bytes``) The code transformer is run after the compilation to bytecode ast_transformer() ------------------ Prototype:: def ast_transformer(tree, context): ... return tree Parameters: * *tree*: an AST tree * *context*: an object with a ``filename`` attribute (``str``) It must return an AST tree. It can modify the AST tree in place, or create a new AST tree. The AST transformer is called after the creation of the AST by the parser and before the compilation to bytecode. New attributes may be added to *context* in the future. Changes ======= In short, add: * ``-o OPTIM_TAG`` command line option * ``ast.Constant`` * ``ast.PyCF_TRANSFORMED_AST`` * ``sys.get_code_transformers()`` * ``sys.implementation.optim_tag`` * ``sys.set_code_transformers(transformers)`` API to get/set code transformers -------------------------------- Add new functions to register code transformers: * ``sys.set_code_transformers(transformers)``: set the list of code transformers and update ``sys.implementation.optim_tag`` * ``sys.get_code_transformers()``: get the list of code transformers. The order of code transformers matter. Running transformer A and then transformer B can give a different output than running transformer B an then transformer A. Example to prepend a new code transformer:: transformers = sys.get_code_transformers() transformers.insert(0, new_cool_transformer) sys.set_code_transformers(transformers) All AST tranformers are run sequentially (ex: the second transformer gets the input of the first transformer), and then all bytecode transformers are run sequentially. Optimizer tag ------------- Changes: * Add ``sys.implementation.optim_tag`` (``str``): optimization tag. The default optimization tag is ``'opt'``. * Add a new ``-o OPTIM_TAG`` command line option to set ``sys.implementation.optim_tag``. Changes on ``importlib``: * ``importlib`` uses ``sys.implementation.optim_tag`` to build the ``.pyc`` filename to importing modules, instead of always using ``opt``. Remove also the special case for the optimizer level ``0`` with the default optimizer tag ``'opt'`` to simplify the code. * When loading a module, if the ``.pyc`` file is missing but the ``.py`` is available, the ``.py`` is only used if code optimizers have the same optimizer tag than the current tag, otherwise an ``ImportError`` exception is raised. Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can be compiled to import a module:: def transformers_tag(): transformers = sys.get_code_transformers() if not transformers: return 'noopt' return '-'.join(transformer.name for transformer in transformers) def use_py(): return (transformers_tag() == sys.implementation.optim_tag) The order of ``sys.get_code_transformers()`` matter. For example, the ``fat`` transformer followed by the ``pythran`` transformer gives the optimizer tag ``fat-pythran``. The behaviour of the ``importlib`` module is unchanged with the default optimizer tag (``'opt'``). Peephole optimizer ------------------ By default, ``sys.implementation.optim_tag`` is ``opt`` and ``sys.get_code_transformers()`` returns a list of one code transformer: the peephole optimizer (optimize the bytecode). Use ``-o noopt`` to disable the peephole optimizer. In this case, the optimizer tag is ``noopt`` and no code transformer is registered. Using the ``-o opt`` option has not effect. AST enhancements ---------------- Enhancements to simplify the implementation of AST transformers: * Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the transformed AST. ``PyCF_ONLY_AST`` returns the AST before the transformers. * Add ``ast.Constant``: this type is not emited by the compiler, but can be used in an AST transformer to simplify the code. It does not contain line number and column offset informations on tuple or frozenset items. * ``PyCodeObject.co_lnotab``: line number delta becomes signed to support moving instructions (note: need to modify MAGIC_NUMBER in importlib). Implemented in the `issue #26107 <https://bugs.python.org/issue26107>`_ * Enhance the bytecode compiler to support ``tuple`` and ``frozenset`` constants. Currently, ``tuple`` and ``frozenset`` constants are created by the peephole transformer, after the bytecode compilation. * ``marshal`` module: fix serialization of the empty frozenset singleton * update ``Tools/parser/unparse.py`` to support the new ``ast.Constant`` node type Examples ======== .pyc filenames -------------- Example of ``.pyc`` filenames of the ``os`` module. With the default optimizer tag ``'opt'``: =========================== ================== .pyc filename Optimization level =========================== ================== ``os.cpython-36.opt-0.pyc`` 0 ``os.cpython-36.opt-1.pyc`` 1 ``os.cpython-36.opt-2.pyc`` 2 =========================== ================== With the ``'fat'`` optimizer tag: =========================== ================== .pyc filename Optimization level =========================== ================== ``os.cpython-36.fat-0.pyc`` 0 ``os.cpython-36.fat-1.pyc`` 1 ``os.cpython-36.fat-2.pyc`` 2 =========================== ================== Bytecode transformer -------------------- Scary bytecode transformer replacing all strings with ``"Ni! Ni! Ni!"``:: import sys class BytecodeTransformer: name = "knights_who_say_ni" def code_transformer(self, code, consts, names, lnotab, context): consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const for const in consts] return (code, consts, names, lnotab) # replace existing code transformers with the new bytecode transformer sys.set_code_transformers([BytecodeTransformer()]) # execute code which will be transformed by code_transformer() exec("print('Hello World!')") Output:: Ni! Ni! Ni! AST transformer --------------- Similary to the bytecode transformer example, the AST transformer also replaces all strings with ``"Ni! Ni! Ni!"``:: import ast import sys class KnightsWhoSayNi(ast.NodeTransformer): def visit_Str(self, node): node.s = 'Ni! Ni! Ni!' return node class ASTTransformer: name = "knights_who_say_ni" def __init__(self): self.transformer = KnightsWhoSayNi() def ast_transformer(self, tree, context): self.transformer.visit(tree) return tree # replace existing code transformers with the new AST transformer sys.set_code_transformers([ASTTransformer()]) # execute code which will be transformed by ast_transformer() exec("print('Hello World!')") Output:: Ni! Ni! Ni! Other Python implementations ============================ The PEP 511 should be implemented by all Python implementation, but the bytecode and the AST are not standardized. By the way, even between minor version of CPython, there are changes on the AST API. There are differences, but only minor differences. It is quite easy to write an AST transformer which works on Python 2.7 and Python 3.5 for example. Discussion ========== * `[Python-Dev] AST optimizer implemented in Python <https://mail.python.org/pipermail/python-dev/2012-August/121286.html>`_ (August 2012) Prior Art ========= AST optimizers -------------- In 2011, Eugene Toder proposed to rewrite some peephole optimizations in a new AST optimizer: issue #11549, `Build-out an AST optimizer, moving some functionality out of the peephole optimizer <https://bugs.python.org/issue11549>`_. The patch adds ``ast.Lit`` (it was proposed to rename it to ``ast.Literal``). In 2012, Victor Stinner wrote the `astoptimizer <https://bitbucket.org/haypo/astoptimizer/>`_ project, an AST optimizer implementing various optimizations. Most interesting optimizations break the Python semantics since no guard is used to disable optimization if something changes. In 2015, Victor Stinner wrote the `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ project, an AST optimizer specializing functions using guards. The Issue #17515 `"Add sys.setasthook() to allow to use a custom AST" optimizer <https://bugs.python.org/issue17515>`_ was a first attempt of API for code transformers, but specific to AST. Python Preprocessors -------------------- * `MacroPy <https://github.com/lihaoyi/macropy>`_: MacroPy is an implementation of Syntactic Macros in the Python Programming Language. MacroPy provides a mechanism for user-defined functions (macros) to perform transformations on the abstract syntax tree (AST) of a Python program at import time. * `pypreprocessor <https://code.google.com/p/pypreprocessor/>`_: C-style preprocessor directives in Python, like ``#define`` and ``#ifdef`` Bytecode transformers --------------------- * `codetransformer <https://pypi.python.org/pypi/codetransformer>`_: Bytecode transformers for CPython inspired by the ``ast`` module’s ``NodeTransformer``. * `byteplay <http://code.google.com/p/byteplay/>`_: Byteplay lets you convert Python code objects into equivalent objects which are easy to play with, and lets you convert those objects back into living Python code objects. It's useful for applying crazy transformations on Python functions, and is also useful in learning Python byte code intricacies. See `byteplay documentation <http://wiki.python.org/moin/ByteplayDoc>`_. See also: * `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_ Copyright ========= This document has been placed in the public domain.

15 52

Respectively and its unpacking sentence
by Mirmojtaba Gharibi 30 Jan '16

30 Jan '16

Hello, I'm thinking of this idea that we have a pseudo-operator called "Respectively" and shown maybe with ; Some examples first: a;b;c = x1;y1;z1 + x2;y2;z2 is equivalent to a=x1+x2 b=y1+y2 c=z1+z2 So it means for each position in the statement, do something like respectively. It's like what I call a vertical expansion, i.e. running statements one by one. Then there is another unpacking operator which maybe we can show with $ sign and it operates on lists and tuples and creates the "Respectively" version of them. So for instance, vec=[]*10 $vec = $u + $v will add two 10-dimensional vectors to each other and put the result in vec. I think this is a syntax that can make many things more concise plus it makes component wise operation on a list done one by one easy. For example, we can calculate the inner product between two vectors like follows (inner product is the sum of component wise multiplication of two vectors): innerProduct =0 innerProduct += $a * $b which is equivalent to innerProduct=0 for i in range(len(a)): ...innerProduct += a[i]+b[i] For example, let's say we want to apply a function to all element in a list, we can do: f($a) The $ and ; take precedence over anything except (). Also, an important thing is that whenever, we don't have the respectively operator, such as for example in the statement above on the left hand side, we basically use the same variable or value or operator for each statement or you can equivalently think we have repeated that whole thing with ;;;;. Such as: s=0 s;s;s += a;b;c; * d;e;f which result in s being a*d+b,c*e+d*f Also, I didn't spot (at least for now any ambiguity). For example one might think what if we do this recursively, such as in: x;y;z + (a;b;c);(d;e;f);(g;h;i) using the formula above this is equivalent to (x;x;x);(y;y;y);(z;z;z)+(a;b;c);(d;e;f);(g;h;i) if we apply print on the statement above, the result will be: x+a x+b x+c y+d y+e y+f z+g z+h z+i Beware that in all of these ; or $ does not create a new list. Rather, they are like creating new lines in the program and executing those lines one by one( in the case of $, to be more accurate, we create for loops). I'll appreciate your time and looking forward to hearing your thoughts. Cheers, Moj

10 22

PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not
by Victor Stinner 29 Jan '16

29 Jan '16

Hi, Thank you for all feedback on my PEP 511. It looks like the current blocker point is the unclear status of "language extensions": code tranformers which deliberately changes the Python semantics. I would like to discuss how we should register them. I think that the PEP 511 must discuss "language extensions" even if it doesn't have to propose a solution to make their usage easier. It's an obvious usage of code transformers. If possible, I would like to find a compromise to support them, but make it explicit that they change the Python semantics. By the way, I discussed with Joseph Jevnik who wrote codetransformer (bytecode transformer) and lazy_python (AST transformer). He wrote me: "One concern that I have though is that transformers are registered globally. I think that the decorators in codetransformer do a good job of signalling to reader the scope of some new code generation." Currently, the PEP 511 doesn't provide a way to register a code transformer but only use it under some conditions. For example, if fatoptimizer is registered, all .pyc files will be called file.cpython-36.fat-0.pyc even if fatoptimizer was disabled. I propose to change the design of sys.set_code_transformers() to use it more like a registry similar to the codecs registry (codecs.register), but different (details below). A difference is that the codecs registry uses a mapping (codec name => codec functions), whereas sys.set_code_transformers() uses an ordered sequence (list) of code transformers. A sequence is used because multiple code transformers can be applied sequentially on a single .py file. Petr Viktorin wrote that language extensions "target specific modules, with which they're closely coupled: The modules won't run without the transformer. And with other modules, the transformer either does nothing (as with MacroPy, hopefully), or would fail altogether (as with Hy). So, they would benefit from specific packages opting in. The effects of enabling them globally range from inefficiency (MacroPy) to failures or needing workarounds (Hy)." Problem (A): solutions proposed below don't make code tranformers mandatory. If a code *requires* a code transformer and the code transformer is not registered, Python doesn't complain. Do you think that it is a real issue in practice? For MacroPy, it's not a problem in practice since functions must be decorated using a decorator from the macropy package. If importing macropy fails, the module cannot be imported. Problem (B): proposed solutions below adds markers to ask to enable a specific code transformer, but a code transformer can decide to always modify the Python semantics without using such marker. According to Nick Coghlan, code transformers changing the Python semantics *must* require a marker in the code using them. IMHO it's the responsability of the author of the code transformer to use markers, not the responsability of Python. Code transformers should maybe return a flag telling if they changed the code or not. I prefer a flag rather than comparing the output to the input, since the comparison can be expensive, especially for a deep AST tree. Example: class Optimizer: def ast_optimizer(self, tree, context): # ... return modified, tree *modified* must be True if tree was modified. There are several options to decide if a code transformer must be used on a specific source file. (1) Add a check_code() and check_ast() functions to code transformers. The code transformer is responsible to decide if it wants to transform the code or not. Python doesn't use the code transformer if the check method returns False. Examples: * MacroPy can search for the "import macropy" statement (of "from macropy import ...") in the AST tree * fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}" in the code: if this variable is found, the optimizer is completly skipped (2) Petr proposed to extend importlib to pass a code transformer when importing a module. importlib.util.import_with_transformer( 'mypackage.specialmodule', MyTransformer()) IMHO this option is too specific: it's restricted to importlib (py_compile, compileall and interactive interpreter don't have the feature). I also dislike the API. (3) Petr also proposed "a special flag in packages": __transformers_for_submodules__ = [MyTransformer()] I don't like having to get access to MyTransformer. The PEP 511 mentions an use case where the transformed code is run *without* registering the transformer. But this issue can easily be fixed by using the string to identify the transformer in the registery (ex: "fat") rather than its class. I'm not sure that putting a flag on the package (package/__init__.py?) is a good idea. I would prefer to enable language extensions on individual files to restrict their scope. (4) Sjoerd Job Postmus proposed something similar but using a comment and not for packages, but any source file: #:Transformers modname.TransformerClassName, modname.OtherTransformerClassName The problem is that comments are not stored in the AST tree. I would prefer to use AST to decide if an AST transformer should be used or not. Note: I'm not really motived to extend the AST to start to include comments, or even code formatting (spaces, newlines, etc.). https://pypi.python.org/pypi/redbaron/ can be used if you want to transform a .py file without touching the format. But I don't think that AST must go to this direction. I prefer to keep AST simple. (5) Nick proposed (indirectly) to use a different filename (don't use ".py") for language extensions. This option works with my option (2): the context contains the filename which can be used to decide to enable or not the code transformer. I understand that the code transformer must also install an importlib hook to search for other filenames than only .py files. Am I right? (6) Nick proposed (indirectly) to use an encoding cookie "which are visible as a comment in the module header". Again, I dislike this option because comments are not stored in AST. Victor

6 7

Re: [Python-ideas] Respectively and its unpacking sentence
by Mirmojtaba Gharibi 28 Jan '16

28 Jan '16

On Thu, Jan 28, 2016 at 3:26 AM, Mirmojtaba Gharibi < mojtaba.gharibi(a)gmail.com> wrote: > > > On Wed, Jan 27, 2016 at 4:15 PM, Andrew Barnert <abarnert(a)yahoo.com> > wrote: > >> On Jan 27, 2016, at 09:55, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >> wrote: >> >> On Wed, Jan 27, 2016 at 2:29 AM, Andrew Barnert <abarnert(a)yahoo.com> >> wrote: >> >>> On Jan 26, 2016, at 22:19, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >>> wrote: >>> >>> Yes, I'm aware sequence unpacking. >>> There is an overlap like you mentioned, but there are things that can't >>> be done with sequence unpacking, but can be done here. >>> >>> For example, let's say you're given two lists that are not necessarily >>> numbers, so you can't use numpy, but you want to apply some component-wise >>> operator between each component. This is something you can't do with >>> sequence unpacking or with numpy. >>> >>> >>> Yes, you can do it with numpy. >>> >>> Obviously you don't get the performance benefits when you aren't using >>> "native" types (like int32) and operations that have vectorizes >>> implementations (like adding two arrays of int32 or taking the dot product >>> of float64 matrices), but you do still get the same elementwise operators, >>> and even a way to apply arbitrary callables over arrays, or even other >>> collections: >>> >>> >>> firsts = ['John', 'Jane'] >>> >>> lasts = ['Smith', 'Doe'] >>> >>> np.vectorize('{1}, {0}'.format)(firsts, lasts) >>> array(['Smith, John', 'Doe, Jane'], dtype='<U11) >>> >>> I think the form I am suggesting is simpler and more readable. >> >> >> But the form you're suggesting doesn't work for vectorizing arbitrary >> functions, only for operator expressions (including simple function calls, >> but that doesn't help for more general function calls). The fact that numpy >> is a little harder to read for cases that your syntax can't handle at all >> is hardly a strike against numpy. >> > > I don't need to vectorize the functions. It's already being done. > Consider the ; example below: > a;b = f(x;y) > it is equivalent to > a=f(x) > b=f(y) > So in effect, in your terminology, it is already vectorized. > Similar example only with $: > a=[0,0,0,0] > x=[1,2,3,4] > $a=f($x) > is equivalent to > a=[0,0,0,0] > x=[1,2,3,4] > for i in range(len(a)): > ...a[i]=f(x[i]) > > > >> >> And, as I already explained, for the cases where your form _does_ work, >> numpy already does it, without all the sigils: >> >> c = a + b >> >> c = a*a + 2*a*b + b*b >> >> c = (a * b).sum() >> >> It also works nicely over multiple dimensions. For example, if a and b >> are both arrays of N 3-vectors instead of just being 3-vectors, you can >> still elementwise-add them just with +; you can sum all of the results with >> sum(axis=1); etc. How would you write any of those things with your >> $-syntax? >> >> I'm happy you brought vectorize to my attention though. I think as soon >> you make the statement just a bit complex, it would become really >> complicated with vectorize. >> >> >> For example lets say you have >> x=[1,2,3,4,5,...] >> y=['A','BB','CCC',...] >> p=[2,3,4,6,6,...] >> r=[]*n >> >> $r = str(len($y*$p)+$x) >> >> >> As a side note, []*n is always just []. Maybe you meant [None for _ in >> range(n)] or [None]*n? Also, where does n come from? It doesn't seem to >> have anything to do with the lengths of x, y, and p. So, what happens if >> it's shorter than them? Or longer? With numpy, of course, that isn't a >> problem--there's no magic being attempted on the = operator (which is good, >> because = isn't an operator in Python, and I'm not sure how you'd even >> properly define your design, much less implement it); the operators just >> create arrays of the right length. >> >> n I just meant symbolically to be len(x). So please replace n with > len(x). I didn't mean to confuse you. sorry. > > >> Anyway, that's still mostly just operators. You _could_ wrap up an >> operator expression in a function to vectorize, but you almost never want >> to. Just use the operators directly on the arrays. >> >> So, let's try a case that has even some minimal amount of logic, where >> translating to operators would be clumsy at best: >> >> @np.vectorize >> def sillyslice(y, x, p): >> if x < p: return y[x:p] >> return y[p:x] >> >> r = sillyslice(y, x, p) >> >> Being a separate function provides all the usual benefits: sillyslice is >> reusable, debuggable, unit-testable, usable as a first-class object, etc. >> But forget that; how would you do this at all with your $-syntax? >> >> Since you didn't answer any of my other questions, I'll snip them and >> repost shorter versions: >> >> * what's wrong with using numpy? Nothing. What's wrong even with for loop >> or assembly for that matter? I didn't argue that it's not possible to >> achieve these things with assembly. >> * what's wrong with APL or J or MATLAB? Not sure how relevant it is to >> our core of conversation. Skipping this. >> * what's wrong with making the operators elementwise instead of wrapping >> the objects in some magic thing? The fact that whenever you >> * what is the type of that magic thing anyway? It has no type. I refer >> you to my very first email. In that email I exactly explained what it >> means. It's at best a psuedo macro or something like that. It exactly is >> equivalent when you write >> > a;b=f(x;y) > to > a=f(x) > b=f(y) > > In other words, if I could interpret my code before python interpreter > interpret it, I would convert the first to the latter. > >

2 1

several different needs [Explicit variable capture list]
by Jim J. Jewett 28 Jan '16

28 Jan '16

I think a small part of the confusion is that there are at least four separate (albeit related) use cases. They all use default arguments for the current workarounds, but they don't have the same ideal solution. (1) Auxiliary variables def f(x, _len=len): ... This is often a micro-optimization; the _len keyword really shouldn't be overridden. Partly because it shouldn't be overridden, having it in the signature is just ugly. This could be solved with another separator in the signature, such as ; or a second () or a new keyword ... def f(x, aux _len=len): ... def f(x, once _len=len): ... def f(x; _len=len):... def f(x)(_len=len): ... def f(x){_len=len}: ... But realistically, that _len isn't ugly *just* because it shouldn't be overridden; it is also inherently ugly. I would prefer that something like Victor's FAT optimizer just make this idiom obsolete. (2) immutable bindings once X final Y const Z This is pretty similar to the auxiliary variables case, except that it tends to be desired more outside of functions. The immutability can be worth documenting on its own, but it feels too much like a typing declaration, which raises questions of "why *this* distinction for *this* variable?" So again, I think something like Victor's FAT optimizer (plus comments when immutability really is important) is a better long-term solution, but I'm not as sure as I was for case 1. (3) Persistent storage def f(x, _cached_results={}): ... In the past, I've managed to convince myself that it is good to be able to pass in a different cache ... or to turn the function into a class, so that I could get to self, or even to attach attributes to the function itself (so that rebinding the name to another function in a naive manner would fail, rather than produces bugs). Those convincings don't stick very well, though. This was clearly at least one of the motivations of some people who asked about static variables. I still think it might be nice to just have a way of easily opening a new scope ... but then I have to explain why I can't just turn the function into a class... So in the end, I suspect this use case is better off ignored, but I am pretty sure it will lead to some extra confusion if any of the others are "solved" in a way that doesn't consider it. (4) Current Value Capture This is the loop variable case that some have thought was the only case under consideration. I don't have anything to add to Andrew Barnert's https://mail.python.org/pipermail/python-ideas/2016-January/037980.html but do see Steven D'Aprano's https://mail.python.org/pipermail/python-ideas/2016-January/038047.html for gotchas even within this use case. -jJ

5 10

intput()
by Marcel O'Neil 27 Jan '16

27 Jan '16

def intput(): return int(input()) Life would be just marginally easier, with a punny function name as a bonus.

12 19

PEP 484 change proposal: Allowing @overload outside stub files
by Guido van Rossum 26 Jan '16

26 Jan '16

Ben Darnell (Tornado lead) brought up a good use case for allowing @overload in regular Python files. There's some discussion (some old, some new) here: https://github.com/ambv/typehinting/issues/72 I now propose to allow @overload in non-stub (i.e. .py) files, but with the following rule: a series of @overload-decorated functions must be followed by an implementation function that's not @overload-decorated. Calling an @overload-decorated function is still an error (I propose NotImplemented). Due to the way repeated function definitions with the same name replace each other, leaving only the last one active, this should work. E.g. for Tornado's utf8() the full definition would look like this: @overloaddef utf8(value: None) -> None: ...@overloaddef utf8(value: bytes) -> bytes: ...@overloaddef utf8(value: str) -> bytes: ... # or (unicode)->bytes, in PY2def utf8(value): # Real implementation goes here. NOTE: If you are trying to understand why we can't use a stub file here or why we can't solve this with type variables or unions, please read the issue and comment there if things are not clear. Here on python-ideas I'd like to focus on seeing whether this amendment is non-controversial (apart from tea party members who just want to repeal PEP 484 entirely :-). I know that previously we wanted to come up with a complete solution for multi-dispatch based on type annotations first, and there are philosophical problems with using @overload (though it can be made to work using sys._getframe()). The proposal here is *not* that solution. If you call the @overload-decorated function, it will raise NotImplemented. (But if you follow the rule, the @overload-decorated function objects are inaccessible so this would only happen if you forgot or misspelled the final, undecorated implementation function). -- --Guido van Rossum (python.org/~guido)

7 11

Making Python great again
by Sven R. Kunze 26 Jan '16

26 Jan '16

Hi, for all those who felt that something is wrong with Python. Here's the solution: https://github.com/samshadwell/TrumpScript ;) Best, Sven

1 0

Documenting asyncio methods as returning awaitables
by Ian Kelly 25 Jan '16

25 Jan '16

The official asyncio documentation includes this note: """ Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be used in a callback-style code, wrap its result with ensure_future(). """ Despite the note, this still causes confusion. See for example https://mail.python.org/pipermail/python-list/2016-January/702342.html As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1, ensure_future is supposed to accept any awaitable. Would it be better then to document these methods as returning awaitables rather than as coroutines?

2 1

Cross link documentation translations
by Julien Palard 25 Jan '16

25 Jan '16

o/ While translating the Python Documentation in French [1][2], I discovered that we're not the only country doing it, there is also Japan [3][4], and Spain [5]. It's possible there's other but I didn't find them (and it's the problem). But there's only a few way for users to find the translations (hearing about them, or explicitly searching for them on a search engine, which they won't do, obviously expecting a link from the english version if they exists). So here is my idea: Why not linking translations from the main documentation? I know that's not directly supported by Sphinx doc [6], but separate sphinx build, blindly (with hardcoded links) linking themselves, may work (like readthedoc is probably doing). The downside of those links is that we'll sometime link to untranslated parts, but those parts may be marked as untranslated [7] to encourage new translators to help. Thoughts? [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] http://docs.python.jp/3/ [4] https://github.com/python-doc-ja/python-doc-ja [5] http://docs.python.org.ar/tutorial/3/index.html [6] https://github.com/sphinx-doc/sphinx/issues/2252 [7] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard

4 4

Multiple dispatch (was Re: PEP 484 change proposal: Allowing <at> overload outside stub files
by Nick Coghlan 24 Jan '16

24 Jan '16

On 24 January 2016 at 07:12, Stefan Krah <skrah.temporarily(a)gmail.com> wrote: > Nick Coghlan <ncoghlan@...> writes: >> You're already going to have to allow this for single lines to handle >> Py2 compatible annotations, so it seems reasonable to also extend it >> to handle overloading while you're still figuring out a native syntax >> for that. > > I find that https://pypi.python.org/pypi/multipledispatch looks quite > nice: > >>>> from multipledispatch import dispatch >>>> @dispatch(int, int) > ... def add(x, y): > ... return x + y > ... >>>> @dispatch(float, float) > ... def add(x, y): > ... return x + y > ... >>>> add(1, 2) > 3 >>>> add(1.0, 2.0) > 3.0 >>>> add(1.0, 2) > Traceback (most recent call last): > File > [cut because gmane.org is inflexible] > line 155, in __call__ > func = self._cache[types] > KeyError: (<class 'float'>, <class 'int'>) Right, the Blaze folks have been doing some very nice work in that area. One of the projects building on multipledispatch is the Odo network of data conversion operations: https://github.com/blaze/odo They do make the somewhat controversial design decision to make dispatch operations process global by default [1], rather than scoping by module. On the other hand, the design also makes it easy to define your own dispatch namespace, so the default orthogonality with the module system likely isn't a problem in practice, and the lack of essential boilerplate does make it very easy to use in contexts like an IPython notebook. There is one aspect that still requires runtime stack introspection [2], and that's getting access to the class scope in order to implicitly make method dispatch specific to the class defining the methods. It's the kind of thing that makes me wonder whether we should be exposing a thread-local variable somewhere with a "class namespace stack" that made it possible to: - tell that you're currently running in the context of a class definition - readily get access to the namespace of the innermost class currently being defined Cheers, Nick. [1] http://multiple-dispatch.readthedocs.org/en/latest/design.html#namespaces-a… [2] https://github.com/mrocklin/multipledispatch/blob/master/multipledispatch/c… -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Proposal to extend PEP 484 (gradual typing) to support Python 2.7
by Agustín Herranz Cecilia 24 Jan '16

24 Jan '16

Hi!, I'd come to this thread late and by coincidence, but after read the whole thread I want to share some thoughts: The main concern it's add a way to add some kind of gradual typing to python 2 code. Because it's working python2 code it can't use annotations and it can't be added in code, so a type comment is the way to go (independent of the convenience, or not, of having type annotation available on runtime). Someone points that using a comment with the python3 annotation signature it's good to educate people on how to use annotations, I feel that's not the point, the point is to people get used to type hints. For this the same syntax must be used across different python versions, so 'function type comments' must be available to use also in python 3 code, this also allow people who can't/won't use annotations to use type hinting. For this, I don't believe that the section "Suggested syntax for Python 2.7 and straddling code" added to PEP 484 is the correct way to go, the proper it's add type comments for functions, as a extension of the "Type comments" section or perhaps in a new PEP. Some concerns that must be take into account to add function type comments: - Using 'type comments' syntax of PEP484 a function signature should look like this: def func(arg1, arg2): # type: Callable[[int, int], int] """ Do something """ return arg1 + arg2 - This easily becomes a long line so breaks PEP8 and linters would complain. So it need to define a way to put the type comment in another line. The type comment will be put In the line after or in the line before? Put in another line will be only available for function type comments or for other type comments too? - With some kind of complex types the type comment surely become a long lint too, how the type comments will be break into different lines? - GVR proposal includes some kind of syntactic sugar for function type comments (" # type: (t_arg1, t_arg2) -> t_ret "). I think it's good but this must be an alternative over typing module syntax (PEP484), not the preferred way (for people get used to typehints). Is this syntactic sugar compatible with generators? The type analyzers could be differentiate between a Callable and a Generator? More concerns on type comments: - As this is intended to gradual type python2 code to port it to python 3 I think it's convenient to add some sort of import that only be used for type checking, and be only imported by the type analyzer, not the runtime. This could be achieve by prepending "#type: " to the normal import statement, something like: # type: import module # type: from package import module - Also there must be addressed how it work on a python2 to python3 environment as there are types with the same name, str for example, that works differently on each python version. If the code is for only one version uses the type names of that version. For 2/3 code types could be define with a "py2" prefix on a module that could be "py2types" having "py2str", for example, to mark things that be of python2 str type. Python 3 types will not have prefixes. I hope this reasoning/ideas will be useful. Also I hope that I have been expressed good enough, English is not my mother tongue. Agustín Herranz.

8 20

Typehinting repo moved to python/typing
by Guido van Rossum 23 Jan '16

23 Jan '16

This is just a note that with Benjamin's help we've moved the ambv/typehinting repo on GitHub into the python org, so its URL is now https://github.com/python/typing . This repo was used most intensely for discussions during PEP 484's drafting period. It also contains the code for typing.py, repackaged for earlier releases on PyPI. The issue tracker is still open for proposals to change PEP 484, which is not unheard of given its provisional status. If you find a pointer to the original location of this repo in a file you can update, please go ahead (though GitHub is pretty good at forwarding URLs from renamed repos). -- --Guido van Rossum (python.org/~guido)

1 0

Fwd: Why do equality tests between OrderedDict keys/values views behave not as expected?
by Alexandre Figura 21 Jan '16

21 Jan '16

Hi, I recently compared OrderedDict instances while writing unit tests, and discovered an interesting behavior. If I create two ordered dictionaries with the same keys/values in the same order, I observe that their values are not equal when I compare them. I recently asked a question about this on Stackoverflow: http://stackoverflow.com/questions/34312674/why-values-of-an-ordereddict-ar… Moreover, another user observed that keys of ordered dictionaries are compared in an order insensitive way: http://stackoverflow.com/questions/34320600/why-does-the-ordereddict-keys-v… Are there any reasons for such implementation choices? As it appears disturbing for many people, would it be possible to update these behaviors? Best Regards, Alexandre.

12 40

Re: [Python-ideas] Boolean value of an Enum member
by Ethan Furman 16 Jan '16

16 Jan '16

[resending to lists -- sorry, Greg] On 01/15/2016 12:36 PM, Greg Ewing wrote: > Ethan Furman wrote: >> So the question now is: for a standard Enum (meaning no other type >> besides Enum is involved) should __bool__ look to the value of the >> Enum member to determine True/False, or should we always be True by >> default and make the Enum creator add their own __bool__ if they want >> something different? > > Can't you just specify a starting value of 0 if you > want the enum to have a false value? That doesn't > seem too onerous to me. You can start with zero, but unless the Enum is mixed with a numeric type it will evaluate to True. Also, but there are other falsey values that a pure Enum member could have: False, None, '', etc., to name a few. However, as Barry said, writing your own is a whopping two lines of code: def __bool__(self): return bool(self._value_) With Barry and Guido's feedback this issue is closed. Thanks everyone! -- ~Ethan~

1 0

Boolean value of an Enum member
by Ethan Furman 16 Jan '16

16 Jan '16

When Enum was being designed one of the questions considered was where to start autonumbering: zero or one. As I remember the discussion we chose not to start with zero because we didn't want an enum member to be False by default, and having a member with value 0 be True was discordant. So the functional API starts with 1 unless overridden. In fact, according to the Enum docs: The reason for defaulting to ``1`` as the starting number and not ``0`` is that ``0`` is ``False`` in a boolean sense, but enum members all evaluate to ``True``. However, if the Enum is combined with some other type (str, int, float, etc), then most behaviour is determined by that type -- including boolean evaluation. So the empty string, 0 values, etc, will cause that Enum member to evaluate as False. So the question now is: for a standard Enum (meaning no other type besides Enum is involved) should __bool__ look to the value of the Enum member to determine True/False, or should we always be True by default and make the Enum creator add their own __bool__ if they want something different? On the one hand we have backwards compatibility, which will take a version to change. On the other hand we have a pretty basic difference in how zero/empty is handled between "pure" Enums and "mixed" Enums. On the gripping hand we have . . . Please respond with your thoughts on changing pure Enums to match mixed Enums or any experience you have had with relying on the "always True" behaviour or if you have implemented your own __bool__ to match the standard True/False meanings or if you have implemented your own __bool__ to match some other scheme entirely. -- ~Ethan~

4 4

(FAT Python) Convert keyword arguments to positional?
by Franklin? Lee 15 Jan '16

15 Jan '16

(FAT Python: http://faster-cpython.readthedocs.org/fat_python.html) FAT Python uses guards to check whether a global name (for example, the name for a function) has changed its value. Idea: If you know exactly which function will be called, you can also optimize based on the properties of that function. According to Eli Bendersky's 2012 blog post[1] (which might be outdated), a function call with keyword arguments is potentially slower than one with only positional arguments. """ If the function in question accepts no arguments (marked by the METH_NOARGS flag when the function is created) or just a single object argument (METH_0 flag), call_function doesn't go through the usual argument packing and can call the underlying function pointer directly. ... do_call ... implements the most generic form of calling. However, there's one more optimization - if func is a PyFunction (an object used internally to represent functions defined in Python code), a separate path is taken - fast_function. ... ... PyCFunction objects that do [receive] keyword arguments [use do_call instead of fast_function]. A curious aspect of this fact is that it's somewhat more efficient to not pass keyword arguments to C functions that either accept them or are fine with just positional arguments. """ So maybe, in a function which uses FAT Python's guards, we can replace some of the keyworded-calls to global function with positional-only calls. It might be a micro-optimization, but it's one that the Python programmer doesn't have to worry about. Concerns: 1. Is it possible to correctly determine, for a given function, which positional parameters have which names? 2. Is it possible to change a function object's named parameters some time after it's created (and inspected)? PS: I didn't feel like this was appropriate for either of Victor's running PEP threads, and the third milestone thread is in the previous month's archives, so I thought that making a new thread would be best. [1] http://eli.thegreenplace.net/2012/03/23/python-internals-how-callables-work

2 2

Password masking for getpass.getpass
by Muhammad Ahmed Khalid 15 Jan '16

15 Jan '16

Greetings, I am working on a project and I am using getpass.getpass() to grab passwords from the user. Some of the users wanted asterisks to be displayed when they were typing in the passwords for feedback i.e. how many characters were typed and how many to backspace. Therefore I have created a solution but I think the feature should come by default (and the programmer should have the option to use a blank or any other character). The code lives here: https://goo.gl/OeY3QI Please let me know about your thoughts on the issue. Also my apologies if this is the wrong mailing list. Please guide me in the right direction if that is the case. Sincerely, King Mak

12 22

Higher leavel heapq
by Joao S. O. Bueno 14 Jan '16

14 Jan '16

Hi, the heapq stdlib module is really handy, but a little low level - in that it accepts a sequence, possibly only a list, as the heap-object, and that object have to be handled independently, outside the functions provided in there. (One can't otherwise insert or delete elements of that list, without destroying the heap, for example). It would be simple to have a higher level class that would do just that, and simplify the use of an ordered container - what about having an extra class there? I have the snippet bellow I wrote on stack-overflow a couple years ago - it is very handy.With a little more boiler plate and code hardening, maybe it could be a nice thing for the stdlib? What do you say? http://stackoverflow.com/questions/8875706/python-heapq-with-custom-compare… js -><-

3 2

Fwd: Using functools.lru_cache only on some arguments of a function
by Bill Winslow 13 Jan '16

13 Jan '16

Sorry for the late reply everyone. I think relying on closures, while a solution, is messy. I'd still much prefer a way to tell lru_cache to merely ignore certain arguments. I'll use some variant of the storing-the-partial-progress-as-an-attribute-on-the-cached-recursive-function method (either with the cached-recursive hidden from the top level via the try/except stuff, or with a more simple wrapper function). I've further considered my original proposal, and rather than naming it "arg_filter", I realized that builtins like sorted(), min(), max(), etc all already have the exact same thing -- a "key" argument which transforms the elements to the user's purpose. (In the sorted/min/max case, it's called on the elements of the argument rather than the argument itself, but it's still the same concept.) So basically, my original proposal with renaming from arg_filter to key, is tantamount to extending the same functionality from sorted/min/max to lru_cache as well. As has been pointed out, my own use case is almost certainly *not* the only use case. The implementation and interface are both simple, and simpler than the alternatives which I'll rely on for now (wrappers and closures, or worse, global singletons etc). I would still like to see it in the stdlib in the future. I've appended a largely similar patch with the proposed additions (there's some internal variable renaming to avoid confusion, resulting in a longer diff). Thanks again for all the input. -Bill ----------------------------------------------------------------------------------------------------------------- https://hg.python.org/cpython/file/3.5/Lib/functools.py diff functools.py.orig functools.py 363c363 < def _make_key(args, kwds, typed, --- > def _make_key(args, kwds, typed, key, 377c377,379 < key = args --- > if key is not None: > args, kwds = key(args, kwds) > cache_key = args 380c382 < key += kwd_mark --- > cache_key += kwd_mark 382c384 < key += item --- > cache_key += item 384c386 < key += tuple(type(v) for v in args) --- > cache_key += tuple(type(v) for v in args) 386,389c388,391 < key += tuple(type(v) for k, v in sorted_items) < elif len(key) == 1 and type(key[0]) in fasttypes: < return key[0] < return _HashedSeq(key) --- > cache_key += tuple(type(v) for k, v in sorted_items) > elif len(cache_key) == 1 and type(cache_key[0]) in fasttypes: > return cache_key[0] > return _HashedSeq(cache_key) 391c393 < def lru_cache(maxsize=128, typed=False): --- > def lru_cache(maxsize=128, typed=False, key=None): 400a403,407 > If *key* is not None, it must be a callable which acts on the arguments > passed to the function. Its return value is used in place of the actual > arguments. It works analogously to the *key* argument to the builtins > sorted, max, and min. > 421a429,431 > if key is not None and not callable(key): > raise TypeErrpr('Expected key to be a callable') > 423c433,434 < wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) --- > wrapper = _lru_cache_wrapper(user_function, maxsize, typed, key, > _CacheInfo) 428c439 < def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo): --- > def _lru_cache_wrapper(user_function, maxsize, typed, key, _CacheInfo): 456,457c467,468 < key = make_key(args, kwds, typed) < result = cache_get(key, sentinel) --- > cache_key = make_key(args, kwds, typed, key) > result = cache_get(cache_key, sentinel) 462c473 < cache[key] = result --- > cache[cache_key] = result 471c482 < key = make_key(args, kwds, typed) --- > cache_key = make_key(args, kwds, typed, key) 473c484 < link = cache_get(key) --- > link = cache_get(cache_key) 487c498 < if key in cache: --- > if cache_key in cache: 496c507 < oldroot[KEY] = key --- > oldroot[KEY] = cache_key 513c524 < cache[key] = oldroot --- > cache[cache_key] = oldroot 517,518c528,529 < link = [last, root, key, result] < last[NEXT] = root[PREV] = cache[key] = link --- > link = [last, root, cache_key, result] > last[NEXT] = root[PREV] = cache[cache_key] = link On Wed, Dec 30, 2015 at 11:10 PM, Michael Selik <mike(a)selik.org> wrote: > On Tue, Dec 29, 2015 at 2:14 AM Franklin? Lee < > leewangzhong+python(a)gmail.com> wrote: > >> On Sat, Dec 12, 2015 at 1:34 PM, Michael Selik <mike(a)selik.org> wrote: >> > On Fri, Dec 11, 2015, 8:20 PM Franklin? Lee < >> leewangzhong+python(a)gmail.com> >> > wrote: >> > > This whole thing is probably best implemented as two separate functions >> > rather than using a closure, depending on how intertwined the code >> paths are >> > for the shortcut/non-shortcut versions. >> >> I like the closure because it has semantic ownership: the inner >> function is a worker for the outer function. >> > > True, a closure has better encapsulation, making it less likely someone > will misuse the helper function. On the other hand, that means there's less > modularity and it would be difficult for someone to use the inner function. > It's hard to know the right choice without seeing the exact problem the > original author was working on. > > >> >> On Fri, Dec 11, 2015 at 8:01 PM, Franklin? Lee >> >> <leewangzhong+python(a)gmail.com> wrote: >> >> > 1. Rewrite your recursive function so that the partial state is a >> >> > nonlocal variable (in the closure), and memoize the recursive part. >> > >> > I'd flip the rare-case to the except block and put the normal-case in >> the >> > try block. I believe this will be more compute-efficient and more >> readable. >> >> The rare case is in the except block, though. >> > > You're correct. Sorry, I somehow misinterpreted the comment, "# To trigger > the exception the first time" as indicating that code path would run only > once. >

3 4

Proposal to extend PEP 484 (gradual typing) to support Python 2.7
by Guido van Rossum 13 Jan '16

13 Jan '16

At Dropbox we're trying to be good citizens and we're working towards introducing gradual typing (PEP 484) into our Python code bases (several million lines of code). However, that code base is mostly still Python 2.7 and we believe that we should introduce gradual typing first and start working on conversion to Python 3 second (since having static types in the code can help a big refactoring like that). Since Python 2 doesn't support function annotations we've had to look for alternatives. We considered stub files, a magic codec, docstrings, and additional `# type:` comments. In the end we decided that `# type:` comments are the most robust approach. We've experimented a fair amount with this and we have a proposal for a standard. The proposal is very simple. Consider the following function with Python 3 annotations: def embezzle(self, account: str, funds: int = 1000000, *fake_receipts: str) -> None: """Embezzle funds from account using fake receipts.""" <code goes here> An equivalent way to write this in Python 2 is the following: def embezzle(self, account, funds=1000000, *fake_receipts): # type: (str, int, *str) -> None """Embezzle funds from account using fake receipts.""" <code goes here> There are a few details to discuss: - Every argument must be accounted for, except 'self' (for instance methods) or 'cls' (for class methods). Also the return type is mandatory. If in Python 3 you would omit some argument or the return type, the Python 2 notation should use 'Any'. - If you're using names defined in the typing module, you must still import them! (There's a backport on PyPI.) - For `*args` and `**kwds`, put 1 or 2 starts in front of the corresponding type annotation. As with Python 3 annotations, the annotation here denotes the type of the individual argument values, not of the tuple/dict that you receive as the special argument value 'args' or 'kwds'. - The entire annotation must be one line. (However, see https://github.com/JukkaL/mypy/issues/1102.) We would like to propose this as a standard (either to be added to PEP 484 or as a new PEP) rather than making it a "proprietary" extension to mypy only, so that others in a similar situation can also benefit. A brief discussion of the considered alternatives: - Stub files: this would complicate the analysis in mypy quite a bit, because it would have to parse both the .py file and the .pyi file and somehow combine the information gathered from both, and for each function it would have to use the types from the stub file to type-check the body of the function in the .py file. This would require a lot of additional plumbing. And if we were using Python 3 we would want to use in-line annotations anyway. - A magic codec was implemented over a year ago ( https://github.com/JukkaL/mypy/tree/master/mypy/codec) but after using it for a bit we didn't like it much. It slows down imports, it requires a `# coding: mypy` declaration, it would conflict with pyxl ( https://github.com/dropbox/pyxl), things go horribly wrong when the codec isn't installed and registered, other tools would be confused by the Python 3 syntax in Python 2 source code, and because of the way the codec was implemented the Python interpreter would occasionally spit out confusing error messages showing the codec's output (which is pretty bare-bones). - While there are existing conventions for specifying types in docstrings, we haven't been using any of these conventions (at least not consistently, nor at an appreciable scale), and they are much more verbose if all you want is adding argument annotations. We're working on a tool that automatically adds type annotations[1], and such a tool would be complicated by the need to integrate the generated annotations into existing docstrings (which, in our code base, unfortunately are wildly incongruous in their conventions). - Finally, the proposed comment syntax is easy to mechanically translate into standard Python 3 function annotations once we're ready to let go of Python 2.7. __________ [1] I have a prototype of such a tool, i mplemented as a 2to3 fixer. It's a bit over 200 lines. It's not very interesting yet, since it sets the types of nearly all arguments to 'Any'. We're considering building a much more advanced version that tries to guess much better argument types using some form of whole-program analysis. I've heard that Facebook's Hack project got a lot of mileage out of such a tool. I don't yet know how to write it yet -- possibly we could use a variant of mypy's type inference engine, or alternatively we might be able to use something like Jedi ( https://github.com/davidhalter/jedi). -- --Guido van Rossum (python.org/~guido)

14 29

RFC: PEP: Specialized functions with guards
by Victor Stinner 13 Jan '16

13 Jan '16

Hi, Here is the second PEP, part of a serie of 3 PEP to add an API to implement a static Python optimizer specializing functions with guards. HTML version: https://faster-cpython.readthedocs.org/pep_specialize.html PEP: xxx Title: Specialized functions with guards Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Add an API to add specialized functions with guards to functions, to support static optimizers respecting the Python semantic. Rationale ========= Python is hard to optimize because almost everything is mutable: builtin functions, function code, global variables, local variables, ... can be modified at runtime. Implement optimizations respecting the Python semantic requires to detect when "something changes", we will call these checks "guards". This PEP proposes to add a ``specialize()`` method to functions to add a specialized functions with guards. When the function is called, the specialized function is used if nothing changed, otherwise use the original bytecode. Writing an optimizer is out of the scope of this PEP. Example ======= Using bytecode -------------- Replace ``chr(65)`` with ``"A"``:: import myoptimizer def func(): return chr(65) def fast_func(): return "A" func.specialize(fast_func.__code__, [myoptimizer.GuardBuiltins("chr")]) del fast_func print("func(): %s" % func()) print("#specialized: %s" % len(func.get_specialized())) print() import builtins builtins.chr = lambda obj: "mock" print("func(): %s" % func()) print("#specialized: %s" % len(func.get_specialized())) Output:: func(): A #specialized: 1 func(): mock #specialized: 0 The hypothetical ``myoptimizer.GuardBuiltins("len")`` is a guard on the builtin ``len()`` function and the ``len`` name in the global namespace. The guard fails if the builtin function is replaced or if a ``len`` name is defined in the global namespace. The first call returns directly the string ``"A"``. The second call removes the specialized function because the builtin ``chr()`` function was replaced, and executes the original bytecode On a microbenchmark, calling the specialized function takes 88 ns, whereas the original bytecode takes 145 ns (+57 ns): 1.6 times as fast. Using builtin function ---------------------- Replace a slow Python function calling ``chr(obj)`` with a direct call to the builtin ``chr()`` function:: import myoptimizer def func(arg): return chr(arg) func.specialize(chr, [myoptimizer.GuardBuiltins("chr")]) print("func(65): %s" % func(65)) print("#specialized: %s" % len(func.get_specialized())) print() import builtins builtins.chr = lambda obj: "mock" print("func(65): %s" % func(65)) print("#specialized: %s" % len(func.get_specialized())) Output:: func(): A #specialized: 1 func(): mock #specialized: 0 The first call returns directly the builtin ``chr()`` function (without creating a Python frame). The second call removes the specialized function because the builtin ``chr()`` function was replaced, and executes the original bytecode. On a microbenchmark, calling the specialized function takes 95 ns, whereas the original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling directly ``chr(65)`` takes 76 ns. Python Function Call ==================== Pseudo-code to call a Python function having specialized functions with guards:: def call_func(func, *args, **kwargs): # by default, call the regular bytecode code = func.__code__.co_code specialized = func.get_specialized() nspecialized = len(specialized) index = 0 while index < nspecialized: guard = specialized[index].guard # pass arguments, some guards need them check = guard(args, kwargs) if check == 1: # guard succeeded: we can use the specialized function code = specialized[index].code break elif check == -1: # guard will always fail: remove the specialized function del specialized[index] elif check == 0: # guard failed temporarely index += 1 # code can be a code object or any callable object execute_code(code, args, kwargs) Changes ======= * Add two new methods to functions: - ``specialize(code, guards: list)``: add specialized function with guard. `code` is a code object (ex: ``func2.__code__``) or any callable object (ex: ``len``). The specialization can be ignored if a guard already fails. - ``get_specialized()``: get the list of specialized functions with guards * Base ``Guard`` type which can be used as parent type to implement guards. It requires to implement a ``check()`` function, with an optional ``first_check()`` function. API: * ``int check(PyObject *guard, PyObject **stack)``: return 1 on success, 0 if the guard failed temporarely, -1 if the guard will always fail * ``int first_check(PyObject *guard, PyObject *func)``: return 0 on success, -1 if the guard will always fail Microbenchmark on ``python3.6 -m timeit -s 'def f(): pass' 'f()'`` (best of 3 runs): * Original Python: 79 ns * Patched Python: 79 ns According to this microbenchmark, the changes has no overhead on calling a Python function without specialization. Behaviour ========= When a function code is replaced (``func.__code__ = new_code``), all specialized functions are removed. When a function is serialized (by ``marshal`` or ``pickle`` for example), specialized functions and guards are ignored (not serialized). Copyright ========= This document has been placed in the public domain. -- Victor

5 10

RFC: PEP: Add dict.__version__
by Victor Stinner 13 Jan '16

13 Jan '16

Hi, Here is a first PEP, part of a serie of 3 PEP to add an API to implement a static Python optimizer specializing functions with guards. HTML version: https://faster-cpython.readthedocs.org/pep_dict_version.html#pep-dict-versi… PEP: xxx Title: Add dict.__version__ Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Add a new read-only ``__version__`` property to ``dict`` and ``collections.UserDict`` types, incremented at each change. Rationale ========= In Python, the builtin ``dict`` type is used by many instructions. For example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the global namespace, or in the builtins namespace (two dict lookups). Python uses ``dict`` for the builtins namespace, globals namespace, type namespaces, instance namespaces, etc. The local namespace (namespace of a function) is usually optimized to an array, but it can be a dict too. Python is hard to optimize because almost everything is mutable: builtin functions, function code, global variables, local variables, ... can be modified at runtime. Implementing optimizations respecting the Python semantic requires to detect when "something changes": we will call these checks "guards". The speedup of optimizations depends on the speed of guard checks. This PEP proposes to add a version to dictionaries to implement efficient guards on namespaces. Example of optimization: replace loading a global variable with a constant. This optimization requires a guard on the global variable to check if it was modified. If the variable is modified, the variable must be loaded at runtime, instead of using the constant. Guard example ============= Pseudo-code of an efficient guard to check if a dictionary key was modified (created, updated or deleted):: UNSET = object() class Guard: def __init__(self, dict, key): self.dict = dict self.key = key self.value = dict.get(key, UNSET) self.version = dict.__version__ def check(self): """Return True if the dictionary value did not changed.""" version = self.dict.__version__ if version == self.version: # Fast-path: avoid the dictionary lookup return True value = self.dict.get(self.key, UNSET) if value == self.value: # another key was modified: # cache the new dictionary version self.version = version return True return False Changes ======= Add a read-only ``__version__`` property to builtin ``dict`` type and to the ``collections.UserDict`` type. New empty dictionaries are initilized to version ``0``. The version is incremented at each change: * ``clear()`` if the dict was non-empty * ``pop(key)`` if the key exists * ``popitem()`` if the dict is non-empty * ``setdefault(key, value)`` if the `key` does not exist * ``__detitem__(key)`` if the key exists * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value is different * ``update(...)`` if new values are different than existing values (the version can be incremented multiple times) Example:: >>> d = {} >>> d.__version__ 0 >>> d['key'] = 'value' >>> d.__version__ 1 >>> d['key'] = 'new value' >>> d.__version__ 2 >>> del d['key'] >>> d.__version__ 3 If a dictionary is created with items, the version is also incremented at each dictionary insertion. Example:: >>> d=dict(x=7, y=33) >>> d.__version__ 2 The version is not incremented is an existing key is modified to the same value, but only the identifier of the value is tested, not the content of the value. Example:: >>> d={} >>> value = object() >>> d['key'] = value >>> d.__version__ 2 >>> d['key'] = value >>> d.__version__ 2 .. note:: CPython uses some singleton like integers in the range [-5; 257], empty tuple, empty strings, Unicode strings of a single character in the range [U+0000; U+00FF], etc. When a key is set twice to the same singleton, the version is not modified. The PEP is designed to implement guards on namespaces, only the ``dict`` type can be used for namespaces in practice. ``collections.UserDict`` is modified because it must mimicks ``dict``. ``collections.Mapping`` is unchanged. Integer overflow ================ The implementation uses the C unsigned integer type ``size_t`` to store the version. On 32-bit systems, the maximum version is ``2**32-1`` (more than ``4.2 * 10 ** 9``, 4 billions). On 64-bit systems, the maximum version is ``2**64-1`` (more than ``1.8 * 10**19``). The C code uses ``version++``. The behaviour on integer overflow of the version is undefined. The minimum guarantee is that the version always changes when the dictionary is modified. The check ``dict.__version__ == old_version`` can be true after an integer overflow, so a guard can return false even if the value changed, which is wrong. The bug occurs if the dict is modified at least ``2**64`` times (on 64-bit system) between two checks of the guard. Using a more complex type (ex: ``PyLongObject``) to avoid the overflow would slow down operations on the ``dict`` type. Even if there is a theorical risk of missing a value change, the risk is considered too low compared to the slow down of using a more complex type. Alternatives ============ Add a version to each dict entry -------------------------------- A single version per dictionary requires to keep a strong reference to the value which can keep the value alive longer than expected. If we add also a version per dictionary entry, the guard can rely on the entry version and so avoid the strong reference to the value (only strong references to a dictionary and key are needed). Changes: add a ``getversion(key)`` method to dictionary which returns ``None`` if the key doesn't exist. When a key is created or modified, the entry version is set to the dictionary version which is incremented at each change (create, modify, delete). Pseudo-code of an efficient guard to check if a dict key was modified using ``getversion()``:: UNSET = object() class Guard: def __init__(self, dict, key): self.dict = dict self.key = key self.dict_version = dict.__version__ self.entry_version = dict.getversion(key) def check(self): """Return True if the dictionary value did not changed.""" dict_version = self.dict.__version__ if dict_version == self.version: # Fast-path: avoid the dictionary lookup return True # lookup in the dictionary, but get the entry version, #not the value entry_version = self.dict.getversion(self.key) if entry_version == self.entry_version: # another key was modified: # cache the new dictionary version self.dict_version = dict_version return True return False This main drawback of this option is the impact on the memory footprint. It increases the size of each dictionary entry, so the overhead depends on the number of buckets (dictionary entries, used or unused yet). For example, it increases the size of each dictionary entry by 8 bytes on 64-bit system if we use ``size_t``. In Python, the memory footprint matters and the trend is more to reduce it. Examples: * `PEP 393 -- Flexible String Representation <https://www.python.org/dev/peps/pep-0393/>`_ * `PEP 412 -- Key-Sharing Dictionary <https://www.python.org/dev/peps/pep-0412/>`_ Add a new dict subtype ---------------------- Add a new ``verdict`` type, subtype of ``dict``. When guards are needed, use the ``verdict`` for namespaces (module namespace, type namespace, instance namespace, etc.) instead of ``dict``. Leave the ``dict`` type unchanged to not add any overhead (memory footprint) when guards are not needed. Technical issue: a lot of C code in the wild, including CPython core, expect the exact ``dict`` type. Issues: * ``exec()`` requires a ``dict`` for globals and locals. A lot of code use ``globals={}``. It is not possible to cast the ``dict`` to a ``dict`` subtype because the caller expects the ``globals`` parameter to be modified (``dict`` is mutable). * Functions call directly ``PyDict_xxx()`` functions, instead of calling ``PyObject_xxx()`` if the object is a ``dict`` subtype * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some functions require the exact ``dict`` type. * ``Python/ceval.c`` does not completly supports dict subtypes for namespaces The ``exec()`` issue is a blocker issue. Other issues: * The garbage collector has a special code to "untrack" ``dict`` instances. If a ``dict`` subtype is used for namespaces, the garbage collector may be unable to break some reference cycles. * Some functions have a fast-path for ``dict`` which would not be taken for ``dict`` subtypes, and so it would make Python a little bit slower. Usage of dict.__version__ ========================= astoptimizer of FAT Python -------------------------- The astoptimizer of the FAT Python project implements many optimizations which require guards on namespaces. Examples: * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on ``builtins.__dict__['len']`` and ``globals()['len']`` are required * Loop unrolling: to unroll the loop ``for i in range(...): ...``, guards on ``builtins.__dict__['range']`` and ``globals()['range']`` are required The `FAT Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project is a static optimizer for Python 3.6. Pyjion ------ According of Brett Cannon, one of the two main developers of Pyjion, Pyjion can also benefit from dictionary version to implement optimizations. Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET Core runtime). Unladen Swallow --------------- Even if dictionary version was not explicitly mentionned, optimization globals and builtins lookup was part of the Unladen Swallow plan: "Implement one of the several proposed schemes for speeding lookups of globals and builtins." Source: `Unladen Swallow ProjectPlan <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_. Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler implemented with LLVM. The project stopped in 2011: `Unladen Swallow Retrospective <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_. Prior Art ========= Cached globals+builtins lookup ------------------------------ In 2006, Andrea Griffini proposes a patch implementing a `Cached globals+builtins lookup optimization <https://bugs.python.org/issue1616125>`_. The patch adds a private ``timestamp`` field to dict. See the thread on python-dev: `About dictionary lookup caching <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_. Globals / builtins cache ------------------------ In 2010, Antoine Pitrou proposed a `Globals / builtins cache <http://bugs.python.org/issue10401>`_ which adds a private ``ma_version`` field to the ``dict`` type. The patch adds a "global and builtin cache" to functions and frames, and changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the cache. PySizer ------- `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python, Google Summer of Code 2005 project by Nick Smallbone. This project has a patch for CPython 2.4 which adds ``key_time`` and ``value_time`` fields to dictionary entries. It uses a global process-wide counter for dictionaries, incremented each time that a dictionary is modified. The times are used to decide when child objects first appeared in their parent objects. Copyright ========= This document has been placed in the public domain. -- Victor

16 79

More friendly access to chmod
by Ram Rachum 12 Jan '16

12 Jan '16

Hi everyone, What do you think about enabling a more friendly interface to chmod information in Python? I believe that currently if I want to get chmod information from a file, I need to do this: my_path.stat().st_mode & 0o777 (I'm using `pathlib`.) (If there's a nicer way than this, please let me know.) This sucks. And then the result is then a number, like 511, which you then have to call `oct` on it to get 0o777. I'm not even happy with getting the octal number. For some of us who live and breathe Linux, seeing a number like 0o440 might be crystal-clear, since your mind automatically translates that to the permissions that user/group/others have, but I haven't reached that level. I would really like an object-oriented approach to chmod, like an object which I can ask "Does group have execute permissions?" and say "Please add read permissions to everyone" etc. Just because Linux speaks in code doesn't mean that we need to. And of course, I'd want that on the `pathlib` module so I could do it all on the path object without referencing another module. What do you think? Ram.

9 20

find-like functionality in pathlib
by Ram Rachum 12 Jan '16

12 Jan '16

What do you think about implementing functionality similar to the `find` utility in Linux in the Pathlib module? I wanted this today, I had a script to write to archive a bunch of files from a folder, and I decided to try writing it in Python rather than in Bash. But I needed something stronger than `Path.glob` in order to select the files. I wanted a regular expression. (In this particular case, I wanted to get a list of all the files excluding the `.git` folder and all files inside of it. Thanks, Ram.

15 36

Re: [Python-ideas] PEP 9 - plaintext PEP format - is officially deprecated
by Barry Warsaw 11 Jan '16

11 Jan '16

On Jan 11, 2016, at 03:25 PM, anatoly techtonik wrote: >On Wed, Jan 6, 2016 at 2:49 AM, Barry Warsaw <barry(a)python.org> wrote: > >> reStructuredText is clearly a better format > >Can you expand on that? I use markdown everywhere reST is better than plain text. Markdown is not a PEP format option. >> all recent PEP submissions have been in reST for a while now anyway. > >Is it possible to query exact numbers automatically? Feel free to grep the PEPs hg repo. >What is the tooling support for handling PEP 9 and PEP 12? UTSL. Everything is in the PEPs hg repo. Cheers, -Barry

1 0

Fall back to encoding unicode strings in utf-8 if latin-1 fails in http.client
by Emil Stenström 08 Jan '16

08 Jan '16

Hi, I hope python-ideas is the right place to post this, I'm very new to this and appreciate a pointer in the right direction if this is not it. The requests project is getting multiple bug reports about a problem in the stdlib http.client, so I thought I'd raise an issue about it here. The bug reports concern people posting http requests with unicode strings when they should be using utf-8 encoded strings. Since RFC 2616 says latin-1 is the default encoding http.client tries that and fails with a UnicodeEncodeError. My idea is NOT to change from latin-1 to something else, that would break compliance with the spec, but instead catch that exception, and try encoding with utf-8 instead. That would avoid breaking backward compatibility, unless someone specifically relied on that exception, which I think is very unlikely. This is also how other languages http libraries seem to deal with this, sending in unicode just works: In cURL (works fine): curl http://example.com -d "Celebrate 🎉" In Ruby with http.rb (works fine): require 'http' r = HTTP.post("http://example.com", :body => "Celebrate 🎉) In Node with request (works fine): var request = require('request'); request.post({url: 'http://example.com', body: "Celebrate 🎉"}, function (error, response, body) { console.log(body) }) But Python 3 with requests crashes instead: import requests r = requests.post("http://localhost:8000/tag", data="Celebrate 🎉") ...with the following stacktrace: ... File "../lib/python3.4/http/client.py", line 1127, in _send_request body = body.encode('iso-8859-1') UnicodeEncodeError: 'latin-1' codec can't encode characters in position 14-15: ordinal not in range(256) ---- So the rationale for this idea is: * http.client doesn't work the way beginners expect for very basic usecases (posting unicode strings) * Libraries in other languages behave like beginners expect, which magnifies the problem. * Changing the default latin-1 encoding probably isn't possible, because it would break the spec... * But catching the exception and try encoding in utf-8 instead wouldn't break the spec and solves the problem. ---- Here's a couple of issues where people expect things to work differently: https://github.com/kennethreitz/requests/issues/1926 https://github.com/kennethreitz/requests/issues/2838 https://github.com/kennethreitz/requests/issues/1822 ---- Does this make sense? /Emil

8 22

intuitive timedeltas like in go
by Sven R. Kunze 07 Jan '16

07 Jan '16

Hi, timedelta handling always felt cumbersome to me: from datetime import timedelta short_period = timedelta(seconds=10) long_period = timedelta(hours=4, seconds=37) Today, I came across this one https://github.com/lxc/lxd/pull/1471/files and I found the creation of a 10 seconds timeout extremely intuitive. Would this represent a valuable addition to Python? from datetime import second, hour short period = 10*second long_period = 4*hour + 37*second Best, Sven

4 3

PEP 9 - plaintext PEP format - is officially deprecated
by Barry Warsaw 06 Jan '16

06 Jan '16

I don't think this will be at all controversial. Brett suggested, and there was no disagreement from the PEP editors, that plain text PEPs be deprecated. reStructuredText is clearly a better format, and all recent PEP submissions have been in reST for a while now anyway. I am therefore withdrawing[*] PEP 9 and have made other appropriate changes to make it clear that only PEP 12 format is acceptable going forward. The PEP editors will not be converting the legacy PEPs to reST, nor will we currently be renaming the relevant PEP source files to end with ".rst" since there's too much tooling that would have to change to do so. However, if either task really interests you, please get in touch with the PEP editors. it-only-took-15-years-ly y'rs, -Barry (on behalf of the PEP editors) [*] Status: Withdrawn being about the only currently appropriate resolution status for process PEPs.

1 0

Re: [Python-ideas] Deprecating the old-style sequence protocol
by Guido van Rossum 05 Jan '16

05 Jan '16

[Adding python-ideas back -- I'm not sure why you dropped it but it looks like an oversight, not intentional] On Fri, Jan 1, 2016 at 2:25 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Dec 27, 2015, at 09:04, Guido van Rossum <guido(a)python.org> wrote: > > > If we want some way to turn something that just defines __getitem__ and > __len__ into a proper sequence, it should just be made to inherit from > Sequence, which supplies the default __iter__ and __reversed__. > (Registration is *not* good enough here.) > > So, if I understand correctly, you're hoping that we can first make the > old-style sequence protocol unnecessary, except for backward compatibility, > and then maybe change the docs to only mention it for backward > compatibility, and only then deprecate it? > That sounds about right. > I think it's worth doing those first two steps, but not actually > deprecating it, at least while Python 2.7 is still around; otherwise, for > dual-version code, something like Steven D'Aprano's "Squares" type would > have to copy Indexable from the 3.x stdlib or get it from some third-party > module like six or backports.collections. > Yes, that's fine. Deprecation sometimes just has to take a really long time. > > If we really want a way to turn something that just supports __getitem__ > into an Iterable maybe we can provide an additional ABC for that purpose; > let's call it a HalfSequence until we've come up with a better name. (We > can't use Iterable for this because Iterable should not reference > __getitem__.) > > #25988 (using Nick's name Indexable, and the details from that post). > Oh, interesting. Though I have misgivings about that name. > > I also think it's fine to introduce Reversible as another ABC and > carefully fit it into the existing hierarchy. It should be a one-trick pony > and be another base class for Sequence; it should not have a default > implementation. (But this has been beaten to death in other threads -- it's > time to just file an issue with a patch.) > > #25987. Thanks! -- --Guido van Rossum (python.org/~guido)

4 6

How exactly does from ... import ... work?
by u8y7541 The Awesome Person 05 Jan '16

05 Jan '16

Suppose I have a file called randomFile.py which reads like this: class A: def __init__(self, foo): self.foo = foo self.bar = bar(foo) class B(A): pass class C(B): pass def bar(foo): return foo + 1 Suppose in another file in the same directory, I have another python program. from randomFile import C # some code When C has to be imported, B also has to be imported because it is the parent. Therefore, A also has to be imported. This also results in the function bar being imported. When from ... import ... is called, does Python follow all the references and import everything that is needed, or does it just import the whole namespace (making wildcard imports acceptable :O)? -- -Surya Subbarao

3 2

Bad programming style in decorators?
by u8y7541 The Awesome Person 05 Jan '16

05 Jan '16

In most decorator tutorials, it's taught using functions inside functions. Isn't this inefficient because every time the decorator is called, you're redefining the function, which takes up much more memory? I prefer defining decorators as classes with __call__ overridden. Is there a reason why decorators are taught with functions inside functions? -- -Surya Subbarao

7 16

Re: [Python-ideas] Bad programming style in decorators?
by u8y7541 The Awesome Person 05 Jan '16

05 Jan '16

Yes, I knew this already. This is what I understood when I said I understood. I thought the *decorator* had a function inside it, not the *decorator factory*. So of course the decorator factory is only called twice. On Mon, Jan 4, 2016 at 10:48 AM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > (off-list, because I think this is no longer relevant to suggesting > changes to Python) > > On Sunday, January 3, 2016 10:58 AM, u8y7541 The Awesome Person < > surya.subbarao1(a)gmail.com> wrote: > > > > >Thanks for explaining the differences tho. I got confused between the > decorator and the decorator factory, thinking the decorator had a function > inside it. Sorry :) > > I think you're _still_ confused. Not your fault, because this is confusing > stuff. The decorator actually does (usually) have a function definition in > it too. But the decorated function--the wrapper that it defines--doesn't. > And that's the thing that you usually call a zillion times, not the > decorator or the decorator factory. > > Let's make it concrete and as simple as possible, and then walk through > all the details: > > > def div(id): > def decorator(func): > > @wraps(func) > def wrapper(*args, **kw): > return "<div id='{}'>{}</div>".format(id, func(*args, > **kw)) > return wrapper > return decorator > > @div('eggs') > def eggs(): > return 'eggs' > > @div('cheese') > def cheeses(): > return '<ul><li>gouda</li><li>edam</li></ul>' > > for _ in range(1000000): > print(eggs()) > print(cheeses()) > > When you're importing the module and hit that "@div('eggs')", that calls > the "div" factory. The only other time that happens is at the > "@div('cheese')". So, the factory does of course create a function, but the > factory only gets called twice in your entire program. (Also, the functions > it creates become garbage as soon as they're called, so by the time you get > to the "for" loop, they've both been deleted.) > > When you finish the "def eggs():" or "def cheeses():" statement, the > decorator function "decorator" returned by "div('eggs')" or "div('cheese')" > gets called. And that decorator also creates a function. But each one only > gets called once in your entire program, and there are only two of them, so > that's only two extra function definitions. (Obviously these two aren't > garbage--they're the functions you call inside the loop.) > > When you hit that "print(eggs())" line, you're calling the decorated > function "wrapper", returned by the decorator function "decorator", > returned by the decorator factory function "div". That function does not > have a function definition inside of it. So, calling it a million times > doesn't cost anything in function definitions. > > And of course "div" itself isn't garbage--you don't need it anymore, but > if you don't tell Python "del div", it'll stick around. So, at your peak, > in the middle of that "for" loop, you have 5 function definitions around > (div, decorated eggs, original eggs, decorated cheese, original cheese). > > If you refactor things differently, you could have 5 functions, 2 class > objects, 2 class instances (your intended class-style design); or 4 > functions, 1 class object, 2 class instances, 2 bound methods, (a simple > class-style decorator); 4 functions, 1 class object, 2 class instances, > (the smallest possible class-style decorator); 2 functions but with > duplicated code and string constants (by inlining div directly into each > function); or 3 functions (with eggs and cheese explicitly calling div--I > suspect this would be actually be smallest here); etc. The difference is > going to be a few hundred bytes one way or the other, and the smallest > possible design may have a severe cost in (time) performance or in > readability. But, as you suggested, and Nick confirmed, there are cases > where it matters. Which means it's worth knowing how to write all the > different possibilities, and evaluate them analytically, and test them. > -- -Surya Subbarao

1 0

Where to put non-collection ABCs (was: Deprecating the old-style sequence protocol)
by Brett Cannon 01 Jan '16

01 Jan '16

Speaking of using ABCs more, where should we put ABCs which have nothing to do with collections? As of right now all ABCs seem to get shoved into collections.abc, but e.g. Awaitable and Coroutine are not types of collections. I personally want to add a context manager ABC with a default __exit__. I opened http://bugs.python.org/issue25637 to discuss this, but I figured a wider discussion wouldn't hurt. Some suggest just putting the ABCs into the abc module. We could create an interfaces module (top-level or a submodule of ABC). The other option is to put the ABCs in subject-specific modules, so my context manager one would go into contextlib (either top-level or an abc submodule); don't know where the coroutine ones would go since it might be overloading asyncio if we out them there. Anyway, the key point is collections.abc is starting to get non-collections stuff and if we are going to start pushing ABCs more we should decide how we want to organize them in general in the stdlib and instead of dumping them into collections.abc. On Sun, Dec 27, 2015, 09:35 Guido van Rossum <guido(a)python.org> wrote: > I think there's a lot of interesting stuff in this thread. Personally I > don't think we should strive to distinguish between mappings and sequences > structurally. We should instead continue to encourage inheriting from (or > registering with) the corresponding ABCs. The goal is to ensure that > there's one best-practice way to distinguish mappings from sequences, and > it's by using isinstance(x, Sequence) or isinstance(x, Mapping). > > If we want some way to turn something that just defines __getitem__ and > __len__ into a proper sequence, it should just be made to inherit from > Sequence, which supplies the default __iter__ and __reversed__. > (Registration is *not* good enough here.) If we really want a way to turn > something that just supports __getitem__ into an Iterable maybe we can > provide an additional ABC for that purpose; let's call it a HalfSequence > until we've come up with a better name. (We can't use Iterable for this > because Iterable should not reference __getitem__.) > > I also think it's fine to introduce Reversible as another ABC and > carefully fit it into the existing hierarchy. It should be a one-trick pony > and be another base class for Sequence; it should not have a default > implementation. (But this has been beaten to death in other threads -- it's > time to just file an issue with a patch.) > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/

6 9