Mailman 3 January 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

5 13

solving multi-core Python
by Eric Snow June 3, 2016

June 3, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort.

26 130

Prevent importing yourself?
by Ned Batchelder March 1, 2016

March 1, 2016

Hi, A common question we get in the #python IRC channel is, "I tried importing a module, but I get an AttributeError trying to use the things it said it provided." Turns out the beginner named their own file the same as the module they were trying to use. That is, they want to try (for example) the "azure" package. So they make a file called azure.py, and start with "import azure". The import succeeds, but it has none of the contents the documentation claims, because they have imported themselves. It's baffling, because they have used the exact statements shown in the examples, but it doesn't work. Could we make this a more obvious failure? Is there ever a valid reason for a file to import itself? Is this situation detectable in the import machinery? --Ned.

14 24

Explicit variable capture list
by haael＠interia.pl Feb. 17, 2016

Feb. 17, 2016

Hi C++ has a nice feature of explicit variable capture list for lambdas: int a = 1, b = 2, c = 3; auto fun = [a, b, c](int x, int y){ return a + b + c + x + y}; This allows easy construction of closures. In Python to achieve that, you need to say: def make_closure(a, b, c): def fun(x, y): return a + b + c + x + y return def a = 1 b = 2 c = 3 fun = make_closure(a, b, c) My proposal: create a special variable qualifier (like global and nonlocal) to automatically capture variables a = 1 b = 2 c = 3 def fun(x, y): capture a, b, c return a + b + c + x + y This will have an effect that symbols a, b and c in the body of the function have values as they had at the moment of function creation. The variables a, b, c must be defined at the time of function creation. If they are not, an error is thrown. The 'capture' qualifier may be combined with keywords global and nonlocal to change lookup behaviour. To make it more useful, we also need some syntax for inline lambdas. I.e.: a = 1 b = 2 c = 3 fun = lambda[a, b, c] x, y: a + b + c + x + y Thanks, haael

28 96

OrderedCounter and OrderedDefaultDict
by Ian Foote Feb. 12, 2016

Feb. 12, 2016

I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: from collections import Counter, OrderedDict class OrderedCounter(Counter, OrderedDict): 'Counter that remembers the order elements are first seen' def __repr__(self): return '%s(%r)' % (self.__class__.__name__, OrderedDict(self)) def __reduce__(self): return self.__class__, (OrderedDict(self),) from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict… which suggests: |fromcollections importOrderedDict,CallableclassDefaultOrderedDict(OrderedDict):# Source: http://stackoverflow.com/a/6190500/562769def__init__(self,default_factory=N… isnotNoneandnotisinstance(default_factory,Callable)):raiseTypeError('first argument must be callable')OrderedDict.__init__(self,*a,**kw)self.default_factory =default_factory def__getitem__(self,key):try:returnOrderedDict.__getitem__(self,key)exceptKeyError:returnself.__missing__(key)def__missing__(self,key):ifself.default_factory isNone:raiseKeyError(key)self[key]=value =self.default_factory()returnvalue def__reduce__(self):ifself.default_factory isNone:args =tuple()else:args =self.default_factory,returntype(self),args,None,None,self.items()defcopy(self):returnself.__copy__()def__copy__(self):returntype(self)(self.default_factory,self)def__deepcopy__(self,memo):importcopy returntype(self)(self.default_factory,copy.deepcopy(self.items()))def__repr__(self):return'OrderedDefaultDict(%s, %s)'%(self.default_factory,OrderedDict.__repr__(self))| This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. Thanks, Ian F

14 23

A bit meta
by Michael Selik Feb. 4, 2016

Feb. 4, 2016

One defect of a mailing list is the difficulty of viewing a weighted average of opinions. The benefit is that anyone can voice an opinion. This is more like the Senate than the House -- Rhode Island appears (on paper) to have as much influence as California. Luckily, we have a form of President. I'm guessing a House occurs in a more private mode of communication? Perhaps as the community gets larger, a system like StackOverflow might be a better tool for handling things like Python-Ideas. > On Jan 27, 2016, at 12:58 PM, Sjoerd Job Postmus <sjoerdjob(a)sjec.nl> wrote: > (not sure if I even have the right to vote here, given that I'm not a > core developer, but just giving my opinion)

28 100

PEP 511: API for code transformers
by Victor Stinner Jan. 30, 2016

Jan. 30, 2016

Hi, This PEP 511 is part of a serie of 3 PEP (509, 510, 511) adding an API to implement a static Python optimizer specializing functions with guards. If the PEP is accepted, it will solve a long list of issues, some issues are old, like #1346238 which is 11 years old ;-) I found 12 issues: * http://bugs.python.org/issue1346238 * http://bugs.python.org/issue2181 * http://bugs.python.org/issue2499 * http://bugs.python.org/issue2506 * http://bugs.python.org/issue4264 * http://bugs.python.org/issue7682 * http://bugs.python.org/issue10399 * http://bugs.python.org/issue11549 * http://bugs.python.org/issue17068 * http://bugs.python.org/issue17430 * http://bugs.python.org/issue17515 * http://bugs.python.org/issue26107 I worked to make the PEP more generic that "this hook is written for FAT Python". Please read the full PEP to see a long list of existing usages in Python of code transformers. You may read again the discussion which occurred 4 years ago about the same topic: https://mail.python.org/pipermail/python-dev/2012-August/121286.html (the thread starts with an idea of AST optimizer, but is moves quickly to a generic API to transform the code) Thanks to Red Hat for giving me time to experiment on this. Victor HTML version: https://www.python.org/dev/peps/pep-0510/#changes PEP: 511 Title: API for code transformers Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-January-2016 Python-Version: 3.6 Abstract ======== Propose an API to register bytecode and AST transformers. Add also ``-o OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o noopt`` disables the peephole optimizer. Raise an ``ImportError`` exception on import if the ``.pyc`` file is missing and the code transformers required to transform the code are missing. code transformers are not needed code transformed ahead of time (loaded from ``.pyc`` files). Rationale ========= Python does not provide a standard way to transform the code. Projects transforming the code use various hooks. The MacroPy project uses an import hook: it adds its own module finder in ``sys.meta_path`` to hook its AST transformer. Another option is to monkey-patch the builtin ``compile()`` function. There are even more options to hook a code transformer. Python 3.4 added a ``compile_source()`` method to ``importlib.abc.SourceLoader``. But code transformation is wider than just importing modules, see described use cases below. Writing an optimizer or a preprocessor is out of the scope of this PEP. Usage 1: AST optimizer ---------------------- Transforming an Abstract Syntax Tree (AST) is a convenient way to implement an optimizer. It's easier to work on the AST than working on the bytecode, AST contains more information and is more high level. Since the optimization can done ahead of time, complex but slow optimizations can be implemented. Example of optimizations which can be implemented with an AST optimizer: * `Copy propagation <https://en.wikipedia.org/wiki/Copy_propagation>`_: replace ``x=1; y=x`` with ``x=1; y=1`` * `Constant folding <https://en.wikipedia.org/wiki/Constant_folding>`_: replace ``1+1`` with ``2`` * `Dead code elimination <https://en.wikipedia.org/wiki/Dead_code_elimination>`_ Using guards (see the `PEP 510 <https://www.python.org/dev/peps/pep-0510/>`_), it is possible to implement a much wider choice of optimizations. Examples: * Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used as iterable * `Loop unrolling <https://en.wikipedia.org/wiki/Loop_unrolling>`_ * Call pure builtins: replace ``len("abc")`` with ``3`` * Copy used builtin symbols to constants * See also `optimizations implemented in fatoptimizer <https://fatoptimizer.readthedocs.org/en/latest/optimizations.html>`_, a static optimizer for Python 3.6. The following issues can be implemented with an AST optimizer: * `Issue #1346238 <https://bugs.python.org/issue1346238>`_: A constant folding optimization pass for the AST * `Issue #2181 <http://bugs.python.org/issue2181>`_: optimize out local variables at end of function * `Issue #2499 <http://bugs.python.org/issue2499>`_: Fold unary + and not on constants * `Issue #4264 <http://bugs.python.org/issue4264>`_: Patch: optimize code to use LIST_APPEND instead of calling list.append * `Issue #7682 <http://bugs.python.org/issue7682>`_: Optimisation of if with constant expression * `Issue #10399 <https://bugs.python.org/issue10399>`_: AST Optimization: inlining of function calls * `Issue #11549 <http://bugs.python.org/issue11549>`_: Build-out an AST optimizer, moving some functionality out of the peephole optimizer * `Issue #17068 <http://bugs.python.org/issue17068>`_: peephole optimization for constant strings * `Issue #17430 <http://bugs.python.org/issue17430>`_: missed peephole optimization Usage 2: Preprocessor --------------------- A preprocessor can be easily implemented with an AST transformer. A preprocessor has various and different usages. Some examples: * Remove debug code like assertions and logs to make the code faster to run it for production. * `Tail-call Optimization <https://en.wikipedia.org/wiki/Tail_call>`_ * Add profiling code * `Lazy evaluation <https://en.wikipedia.org/wiki/Lazy_evaluation>`_: see `lazy_python <https://github.com/llllllllll/lazy_python>`_ (bytecode transformer) and `lazy macro of MacroPy <https://github.com/lihaoyi/macropy#lazy>`_ (AST transformer) * Change dictionary literals into collection.OrderedDict instances * Declare constants: see `@asconstants of codetransformer <https://pypi.python.org/pypi/codetransformer>`_ * Domain Specific Language (DSL) like SQL queries. The Python language itself doesn't need to be modified. Previous attempts to implement DSL for SQL like `PEP 335 - Overloadable Boolean Operators <https://www.python.org/dev/peps/pep-0335/>`_ was rejected. * Pattern Matching of functional languages * String Interpolation, but `PEP 498 -- Literal String Interpolation <https://www.python.org/dev/peps/pep-0498/>`_ was merged into Python 3.6. `MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of examples and use cases. This PEP does not add any new code transformer. Using a code transformer will require an external module and to register it manually. See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python obfuscator, deobfuscator, and user-assisted decompiler. Usage 3: Disable all optimization --------------------------------- Ned Batchelder asked to add an option to disable the peephole optimizer because it makes code coverage more difficult to implement. See the discussion on the python-ideas mailing list: `Disable all peephole optimizations <https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_. This PEP adds a new ``-o noopt`` command line option to disable the peephole optimizer. In Python, it's as easy as:: sys.set_code_transformers([]) It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add mechanism to disable optimizations. Usage 4: Write new bytecode optimizers in Python ------------------------------------------------ Python 3.6 optimizes the code using a peephole optimizer. By definition, a peephole optimizer has a narrow view of the code and so can only implement basic optimizations. The optimizer rewrites the bytecode. It is difficult to enhance it, because it written in C. With this PEP, it becomes possible to implement a new bytecode optimizer in pure Python and experiment new optimizations. Some optimizations are easier to implement on the AST like constant folding, but optimizations on the bytecode are still useful. For example, when the AST is compiled to bytecode, useless jumps can be emited because the compiler is naive and does not try to optimize anything. Use Cases ========= This section give examples of use cases explaining when and how code transformers will be used. Interactive interpreter ----------------------- It will be possible to use code transformers with the interactive interpreter which is popular in Python and commonly used to demonstrate Python. The code is transformed at runtime and so the interpreter can be slower when expensive code transformers are used. Build a transformed package --------------------------- It will be possible to build a package of the transformed code. A transformer can have a configuration. The configuration is not stored in the package. All ``.pyc`` files of the package must be transformed with the same code transformers and the same transformers configuration. It is possible to build different ``.pyc`` files using different optimizer tags. Example: ``fat`` for the default configuration and ``fat_inline`` for a different configuration with function inlining enabled. A package can contain ``.pyc`` files with different optimizer tags. Install a package containing transformed .pyc files --------------------------------------------------- It will be possible to install a package which contains transformed ``.pyc`` files. All ``.pyc`` files with any optimizer tag contained in the package are installed, not only for the current optimizer tag. Build .pyc files when installing a package ------------------------------------------ If a package does not contain any ``.pyc`` files of the current optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are created during the installation. Code transformers of the optimizer tag are required. Otherwise, the installation fails with an error. Execute transformed code ------------------------ It will be possible to execute transformed code. Raise an ``ImportError`` exception on import if the ``.pyc`` file of the current optimizer tag is missing and the code transformers required to transform the code are missing. The interesting point here is that code transformers are not needed to execute the transformed code if all required ``.pyc`` files are already available. Code transformer API ==================== A code transformer is a class with ``ast_transformer()`` and/or ``code_transformer()`` methods (API described below) and a ``name`` attribute. For efficiency, do not define a ``code_transformer()`` or ``ast_transformer()`` method if it does nothing. The ``name`` attribute (``str``) must be a short string used to identify an optimizer. It is used to build a ``.pyc`` filename. The name must not contain dots (``'.'``), dashes (``'-'``) or directory separators: dots are used to separated fields in a ``.pyc`` filename and dashes areused to join code transformer names to build the optimizer tag. .. note:: It would be nice to pass the fully qualified name of a module in the *context* when an AST transformer is used to transform a module on import, but it looks like the information is not available in ``PyParser_ASTFromStringObject()``. code_transformer() ------------------ Prototype:: def code_transformer(code, consts, names, lnotab, context): ... return (code, consts, names, lnotab) Parameters: * *code*: the bytecode (``bytes``) * *consts*: a sequence of constants * *names*: tuple of variable names * *lnotab*: table mapping instruction offsets to line numbers (``bytes``) The code transformer is run after the compilation to bytecode ast_transformer() ------------------ Prototype:: def ast_transformer(tree, context): ... return tree Parameters: * *tree*: an AST tree * *context*: an object with a ``filename`` attribute (``str``) It must return an AST tree. It can modify the AST tree in place, or create a new AST tree. The AST transformer is called after the creation of the AST by the parser and before the compilation to bytecode. New attributes may be added to *context* in the future. Changes ======= In short, add: * ``-o OPTIM_TAG`` command line option * ``ast.Constant`` * ``ast.PyCF_TRANSFORMED_AST`` * ``sys.get_code_transformers()`` * ``sys.implementation.optim_tag`` * ``sys.set_code_transformers(transformers)`` API to get/set code transformers -------------------------------- Add new functions to register code transformers: * ``sys.set_code_transformers(transformers)``: set the list of code transformers and update ``sys.implementation.optim_tag`` * ``sys.get_code_transformers()``: get the list of code transformers. The order of code transformers matter. Running transformer A and then transformer B can give a different output than running transformer B an then transformer A. Example to prepend a new code transformer:: transformers = sys.get_code_transformers() transformers.insert(0, new_cool_transformer) sys.set_code_transformers(transformers) All AST tranformers are run sequentially (ex: the second transformer gets the input of the first transformer), and then all bytecode transformers are run sequentially. Optimizer tag ------------- Changes: * Add ``sys.implementation.optim_tag`` (``str``): optimization tag. The default optimization tag is ``'opt'``. * Add a new ``-o OPTIM_TAG`` command line option to set ``sys.implementation.optim_tag``. Changes on ``importlib``: * ``importlib`` uses ``sys.implementation.optim_tag`` to build the ``.pyc`` filename to importing modules, instead of always using ``opt``. Remove also the special case for the optimizer level ``0`` with the default optimizer tag ``'opt'`` to simplify the code. * When loading a module, if the ``.pyc`` file is missing but the ``.py`` is available, the ``.py`` is only used if code optimizers have the same optimizer tag than the current tag, otherwise an ``ImportError`` exception is raised. Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can be compiled to import a module:: def transformers_tag(): transformers = sys.get_code_transformers() if not transformers: return 'noopt' return '-'.join(transformer.name for transformer in transformers) def use_py(): return (transformers_tag() == sys.implementation.optim_tag) The order of ``sys.get_code_transformers()`` matter. For example, the ``fat`` transformer followed by the ``pythran`` transformer gives the optimizer tag ``fat-pythran``. The behaviour of the ``importlib`` module is unchanged with the default optimizer tag (``'opt'``). Peephole optimizer ------------------ By default, ``sys.implementation.optim_tag`` is ``opt`` and ``sys.get_code_transformers()`` returns a list of one code transformer: the peephole optimizer (optimize the bytecode). Use ``-o noopt`` to disable the peephole optimizer. In this case, the optimizer tag is ``noopt`` and no code transformer is registered. Using the ``-o opt`` option has not effect. AST enhancements ---------------- Enhancements to simplify the implementation of AST transformers: * Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the transformed AST. ``PyCF_ONLY_AST`` returns the AST before the transformers. * Add ``ast.Constant``: this type is not emited by the compiler, but can be used in an AST transformer to simplify the code. It does not contain line number and column offset informations on tuple or frozenset items. * ``PyCodeObject.co_lnotab``: line number delta becomes signed to support moving instructions (note: need to modify MAGIC_NUMBER in importlib). Implemented in the `issue #26107 <https://bugs.python.org/issue26107>`_ * Enhance the bytecode compiler to support ``tuple`` and ``frozenset`` constants. Currently, ``tuple`` and ``frozenset`` constants are created by the peephole transformer, after the bytecode compilation. * ``marshal`` module: fix serialization of the empty frozenset singleton * update ``Tools/parser/unparse.py`` to support the new ``ast.Constant`` node type Examples ======== .pyc filenames -------------- Example of ``.pyc`` filenames of the ``os`` module. With the default optimizer tag ``'opt'``: =========================== ================== .pyc filename Optimization level =========================== ================== ``os.cpython-36.opt-0.pyc`` 0 ``os.cpython-36.opt-1.pyc`` 1 ``os.cpython-36.opt-2.pyc`` 2 =========================== ================== With the ``'fat'`` optimizer tag: =========================== ================== .pyc filename Optimization level =========================== ================== ``os.cpython-36.fat-0.pyc`` 0 ``os.cpython-36.fat-1.pyc`` 1 ``os.cpython-36.fat-2.pyc`` 2 =========================== ================== Bytecode transformer -------------------- Scary bytecode transformer replacing all strings with ``"Ni! Ni! Ni!"``:: import sys class BytecodeTransformer: name = "knights_who_say_ni" def code_transformer(self, code, consts, names, lnotab, context): consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const for const in consts] return (code, consts, names, lnotab) # replace existing code transformers with the new bytecode transformer sys.set_code_transformers([BytecodeTransformer()]) # execute code which will be transformed by code_transformer() exec("print('Hello World!')") Output:: Ni! Ni! Ni! AST transformer --------------- Similary to the bytecode transformer example, the AST transformer also replaces all strings with ``"Ni! Ni! Ni!"``:: import ast import sys class KnightsWhoSayNi(ast.NodeTransformer): def visit_Str(self, node): node.s = 'Ni! Ni! Ni!' return node class ASTTransformer: name = "knights_who_say_ni" def __init__(self): self.transformer = KnightsWhoSayNi() def ast_transformer(self, tree, context): self.transformer.visit(tree) return tree # replace existing code transformers with the new AST transformer sys.set_code_transformers([ASTTransformer()]) # execute code which will be transformed by ast_transformer() exec("print('Hello World!')") Output:: Ni! Ni! Ni! Other Python implementations ============================ The PEP 511 should be implemented by all Python implementation, but the bytecode and the AST are not standardized. By the way, even between minor version of CPython, there are changes on the AST API. There are differences, but only minor differences. It is quite easy to write an AST transformer which works on Python 2.7 and Python 3.5 for example. Discussion ========== * `[Python-Dev] AST optimizer implemented in Python <https://mail.python.org/pipermail/python-dev/2012-August/121286.html>`_ (August 2012) Prior Art ========= AST optimizers -------------- In 2011, Eugene Toder proposed to rewrite some peephole optimizations in a new AST optimizer: issue #11549, `Build-out an AST optimizer, moving some functionality out of the peephole optimizer <https://bugs.python.org/issue11549>`_. The patch adds ``ast.Lit`` (it was proposed to rename it to ``ast.Literal``). In 2012, Victor Stinner wrote the `astoptimizer <https://bitbucket.org/haypo/astoptimizer/>`_ project, an AST optimizer implementing various optimizations. Most interesting optimizations break the Python semantics since no guard is used to disable optimization if something changes. In 2015, Victor Stinner wrote the `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ project, an AST optimizer specializing functions using guards. The Issue #17515 `"Add sys.setasthook() to allow to use a custom AST" optimizer <https://bugs.python.org/issue17515>`_ was a first attempt of API for code transformers, but specific to AST. Python Preprocessors -------------------- * `MacroPy <https://github.com/lihaoyi/macropy>`_: MacroPy is an implementation of Syntactic Macros in the Python Programming Language. MacroPy provides a mechanism for user-defined functions (macros) to perform transformations on the abstract syntax tree (AST) of a Python program at import time. * `pypreprocessor <https://code.google.com/p/pypreprocessor/>`_: C-style preprocessor directives in Python, like ``#define`` and ``#ifdef`` Bytecode transformers --------------------- * `codetransformer <https://pypi.python.org/pypi/codetransformer>`_: Bytecode transformers for CPython inspired by the ``ast`` module’s ``NodeTransformer``. * `byteplay <http://code.google.com/p/byteplay/>`_: Byteplay lets you convert Python code objects into equivalent objects which are easy to play with, and lets you convert those objects back into living Python code objects. It's useful for applying crazy transformations on Python functions, and is also useful in learning Python byte code intricacies. See `byteplay documentation <http://wiki.python.org/moin/ByteplayDoc>`_. See also: * `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_ Copyright ========= This document has been placed in the public domain.

15 52