Mailman 3 October 2013 - Python-Dev

A C implementation of OrderedDict
by Eric Snow 21 Oct '13

21 Oct '13

If anyone is interested in having a (faithful) C implementation of OrderedDict for the 3.4 release, I have a patch up [1]. My interest in having the patch applied is relative to proposals that won't apply to 3.4, so I haven't felt the need to advance the patch. However, in case anyone else would find it useful for 3.4, I figured I would point it out. While the patch isn't all that complicated, it is large and my C/C-API experience isn't proportional to that size. So I don't feel comfortable about moving ahead with the patch without at least one thorough review. Thanks. -eric [1] http://bugs.python.org/issue16991

2 2

[RELEASED] Python 3.4.0a4
by Larry Hastings 21 Oct '13

21 Oct '13

On behalf of the Python development team, I'm very pleased to announce the fourth and final alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, the new "statistics" module * PEP 3156, the new "asyncio" module, a new framework for asynchronous I/O To download Python 3.4.0a4 visit: http://www.python.org/download/releases/3.4.0/ Python 3.4.0a4 has several known issues: * The Python compiler has a small memory leak. * The "asyncio" module test suite fails on some platforms. * I/O conducted by the "asyncio" module may, rarely, erroneously time out. The timeout takes one hour. Please consider trying Python 3.4.0a4 with your code and reporting any new issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors)

1 0

Logging Module format
by Hasan Diwan 21 Oct '13

21 Oct '13

I've been using the logging module recently and noticed the default format doesn't timestamp log entries. I've not figured out how to change the format after initialization. This is python 2.7, on Mac OS X. Help, anyone? -- H -- Sent from my mobile device Envoyé de mon portable

2 1

Python 3.4.0a4: Coming In For A Bumpy Landing
by Larry Hastings 21 Oct '13

21 Oct '13

3.4.0a4 is tagged and I'm in the process of releasing it. But it's going to be, let's say, more "alpha-quality" than the previous alphas. Known problems: * There's a reference count leak in the compiler. * asyncio test suite sometimes times out, which takes... an hour. * asyncio test suit fails on a couple of platforms. I don't think any of these are bad enough to slip the schedule. However, if you disagree, pipe up, feel free to try and change my mind, it's not too late. //arry/

3 3

PEP 451: ModuleSpec
by Eric Snow 20 Oct '13

20 Oct '13

After a few rounds on import-sig PEP 451 is really for general consumption. I also have a patch up now. HTML: http://www.python.org/dev/peps/pep-0451/ implementation: http://bugs.python.org/issue18864 Your comments would be appreciated. -eric ============================================================= PEP: 451 Title: A ModuleSpec Type for the Import System Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently(a)gmail.com> Discussions-To: import-sig(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Aug-2013 Python-Version: 3.4 Post-History: 8-Aug-2013, 28-Aug-2013, 18-Sep-2013, 24-Sep-2013 Resolution: Abstract ======== This PEP proposes to add a new class to importlib.machinery called "ModuleSpec". It will provide all the import-related information used to load a module and will be available without needing to load the module first. Finders will directly provide a module's spec instead of a loader (which they will continue to provide indirectly). The import machinery will be adjusted to take advantage of module specs, including using them to load modules. Terms and Concepts ================== The changes in this proposal are an opportunity to make several existing terms and concepts more clear, whereas currently they are (unfortunately) ambiguous. New concepts are also introduced in this proposal. Finally, it's worth explaining a few other existing terms with which people may not be so familiar. For the sake of context, here is a brief summary of all three groups of terms and concepts. A more detailed explanation of the import system is found at [import_system_docs]_. finder ------ A "finder" is an object that identifies the loader that the import system should use to load a module. Currently this is accomplished by calling the finder's find_module() method, which returns the loader. Finders are strictly responsible for providing the loader, which they do through their find_module() method. The import system then uses that loader to load the module. loader ------ A "loader" is an object that is used to load a module during import. Currently this is done by calling the loader's load_module() method. A loader may also provide APIs for getting information about the modules it can load, as well as about data from sources associated with such a module. Right now loaders (via load_module()) are responsible for certain boilerplate, import-related operations. These are: 1. perform some (module-related) validation; 2. create the module object; 3. set import-related attributes on the module; 4. "register" the module to sys.modules; 5. exec the module; 6. clean up in the event of failure while loading the module. This all takes place during the import system's call to Loader.load_module(). origin ------ This is a new term and concept. The idea of it exists subtly in the import system already, but this proposal makes the concept explicit. "origin" in an import context means the system (or resource within a system) from which a module originates. For the purposes of this proposal, "origin" is also a string which identifies such a resource or system. "origin" is applicable to all modules. For example, the origin for built-in and frozen modules is the interpreter itself. The import system already identifies this origin as "built-in" and "frozen", respectively. This is demonstrated in the following module repr: "<module 'sys' (built-in)>". In fact, the module repr is already a relatively reliable, though implicit, indicator of a module's origin. Other modules also indicate their origin through other means, as described in the entry for "location". It is up to the loader to decide on how to interpret and use a module's origin, if at all. location -------- This is a new term. However the concept already exists clearly in the import system, as associated with the ``__file__`` and ``__path__`` attributes of modules, as well as the name/term "path" elsewhere. A "location" is a resource or "place", rather than a system at large, from which a module is loaded. It qualifies as an "origin". Examples of locations include filesystem paths and URLs. A location is identified by the name of the resource, but may not necessarily identify the system to which the resource pertains. In such cases the loader would have to identify the system itself. In contrast to other kinds of module origin, a location cannot be inferred by the loader just by the module name. Instead, the loader must be provided with a string to identify the location, usually by the finder that generates the loader. The loader then uses this information to locate the resource from which it will load the module. In theory you could load the module at a given location under various names. The most common example of locations in the import system are the files from which source and extension modules are loaded. For these modules the location is identified by the string in the ``__file__`` attribute. Although ``__file__`` isn't particularly accurate for some modules (e.g. zipped), it is currently the only way that the import system indicates that a module has a location. A module that has a location may be called "locatable". cache ----- The import system stores compiled modules in the __pycache__ directory as an optimization. This module cache that we use today was provided by PEP 3147. For this proposal, the relevant API for module caching is the ``__cache__`` attribute of modules and the cache_from_source() function in importlib.util. Loaders are responsible for putting modules into the cache (and loading out of the cache). Currently the cache is only used for compiled source modules. However, loaders may take advantage of the module cache for other kinds of modules. package ------- The concept does not change, nor does the term. However, the distinction between modules and packages is mostly superficial. Packages *are* modules. They simply have a ``__path__`` attribute and import may add attributes bound to submodules. The typically perceived difference is a source of confusion. This proposal explicitly de-emphasizes the distinction between packages and modules where it makes sense to do so. Motivation ========== The import system has evolved over the lifetime of Python. In late 2002 PEP 302 introduced standardized import hooks via finders and loaders and sys.meta_path. The importlib module, introduced with Python 3.1, now exposes a pure Python implementation of the APIs described by PEP 302, as well as of the full import system. It is now much easier to understand and extend the import system. While a benefit to the Python community, this greater accessibilty also presents a challenge. As more developers come to understand and customize the import system, any weaknesses in the finder and loader APIs will be more impactful. So the sooner we can address any such weaknesses the import system, the better...and there are a couple we can take care of with this proposal. Firstly, any time the import system needs to save information about a module we end up with more attributes on module objects that are generally only meaningful to the import system. It would be nice to have a per-module namespace in which to put future import-related information and to pass around within the import system. Secondly, there's an API void between finders and loaders that causes undue complexity when encountered. The PEP 420 (namespace packages) implementation had to work around this. The complexity surfaced again during recent efforts on a separate proposal. [ref_files_pep]_ The `finder`_ and `loader`_ sections above detail current responsibility of both. Notably, loaders are not required to provide any of the functionality of their load_module() through other methods. Thus, though the import-related information about a module is likely available without loading the module, it is not otherwise exposed. Furthermore, the requirements assocated with load_module() are common to all loaders and mostly are implemented in exactly the same way. This means every loader has to duplicate the same boilerplate code. importlib.util provides some tools that help with this, but it would be more helpful if the import system simply took charge of these responsibilities. The trouble is that this would limit the degree of customization that load_module() could easily continue to facilitate. More importantly, While a finder *could* provide the information that the loader's load_module() would need, it currently has no consistent way to get it to the loader. This is a gap between finders and loaders which this proposal aims to fill. Finally, when the import system calls a finder's find_module(), the finder makes use of a variety of information about the module that is useful outside the context of the method. Currently the options are limited for persisting that per-module information past the method call, since it only returns the loader. Popular options for this limitation are to store the information in a module-to-info mapping somewhere on the finder itself, or store it on the loader. Unfortunately, loaders are not required to be module-specific. On top of that, some of the useful information finders could provide is common to all finders, so ideally the import system could take care of those details. This is the same gap as before between finders and loaders. As an example of complexity attributable to this flaw, the implementation of namespace packages in Python 3.3 (see PEP 420) added FileFinder.find_loader() because there was no good way for find_module() to provide the namespace search locations. The answer to this gap is a ModuleSpec object that contains the per-module information and takes care of the boilerplate functionality involved with loading the module. Specification ============= The goal is to address the gap between finders and loaders while changing as little of their semantics as possible. Though some functionality and information is moved to the new ModuleSpec type, their behavior should remain the same. However, for the sake of clarity the finder and loader semantics will be explicitly identified. Here is a high-level summary of the changes described by this PEP. More detail is available in later sections. importlib.machinery.ModuleSpec (new) ------------------------------------ A specification for a module's import-system-related state. See the `ModuleSpec`_ section below for a more detailed description. * ModuleSpec(name, loader, \*, origin=None, loader_state=None, is_package=None) Attributes: * name - a string for the name of the module. * loader - the loader to use for loading. * origin - the name of the place from which the module is loaded, e.g. "builtin" for built-in modules and the filename for modules loaded from source. * submodule_search_locations - list of strings for where to find submodules, if a package (None otherwise). * loader_state - a container of extra module-specific data for use during loading. * cached (property) - a string for where the compiled module should be stored. * parent (RO-property) - the name of the package to which the module belongs as a submodule (or None). * has_location (RO-property) - a flag indicating whether or not the module's "origin" attribute refers to a location. Instance Methods: * module_repr() - provide a repr string for the spec'ed module; non-locatable modules will use their origin (e.g. "built-in"). * init_module_attrs(module) - set any of a module's import-related attributes that aren't already set. importlib.util Additions ------------------------ These are ModuleSpec factory functions, meant as a convenience for finders. See the `Factory Functions`_ section below for more detail. * spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None) - build a spec from file-oriented information and loader APIs. * from_loader(name, loader, \*, origin=None, is_package=None) - build a spec with missing information filled in by using loader APIs. Other API Additions ------------------- * importlib.find_spec(name, path=None) will work exactly the same as importlib.find_loader() (which it replaces), but return a spec instead of a loader. For loaders: * importlib.abc.Loader.exec_module(module) will execute a module in its own namespace. It replaces importlib.abc.Loader.load_module(), taking over its module execution functionality. * importlib.abc.Loader.create_module(spec) (optional) will return the module to use for loading. For modules: * Module objects will have a new attribute: ``__spec__``. API Changes ----------- * InspectLoader.is_package() will become optional. Deprecations ------------ * importlib.abc.MetaPathFinder.find_module() * importlib.abc.PathEntryFinder.find_module() * importlib.abc.PathEntryFinder.find_loader() * importlib.abc.Loader.load_module() * importlib.abc.Loader.module_repr() * The parameters and attributes of the various loaders in importlib.machinery * importlib.util.set_package() * importlib.util.set_loader() * importlib.find_loader() Removals -------- These were introduced prior to Python 3.4's release, so they can simply be removed. * importlib.abc.Loader.init_module_attrs() * importlib.util.module_to_load() Other Changes ------------- * The import system implementation in importlib will be changed to make use of ModuleSpec. * importlib.reload() will make use of ModuleSpec. * A module's import-related attributes (other than ``__spec__``) will no longer be used directly by the import system during that module's import. However, this does not impact use of those attributes (e.g. ``__path__``) when loading other modules (e.g. submodules). * Import-related attributes should no longer be added to modules directly, except by the import system. * The module type's ``__repr__()`` will be a thin wrapper around a pure Python implementation which will leverage ModuleSpec. * The spec for the ``__main__`` module will reflect the appropriate name and origin. Backward-Compatibility ---------------------- * If a finder does not define find_spec(), a spec is derived from the loader returned by find_module(). * PathEntryFinder.find_loader() still takes priority over find_module(). * Loader.load_module() is used if exec_module() is not defined. What Will not Change? --------------------- * The syntax and semantics of the import statement. * Existing finders and loaders will continue to work normally. * The import-related module attributes will still be initialized with the same information. * Finders will still create loaders (now storing them in specs). * Loader.load_module(), if a module defines it, will have all the same requirements and may still be called directly. * Loaders will still be responsible for module data APIs. * importlib.reload() will still overwrite the import-related attributes. Responsibilities ---------------- Here's a quick breakdown of where responsibilities lie after this PEP. finders: * create loader * create spec loaders: * create module (optional) * execute module ModuleSpec: * orchestrate module loading * boilerplate for module loading, including managing sys.modules and setting import-related attributes * create module if loader doesn't * call loader.exec_module(), passing in the module in which to exec * contain all the information the loader needs to exec the module * provide the repr for modules What Will Existing Finders and Loaders Have to Do Differently? ============================================================== Immediately? Nothing. The status quo will be deprecated, but will continue working. However, here are the things that the authors of finders and loaders should change relative to this PEP: * Implement find_spec() on finders. * Implement exec_module() on loaders, if possible. The ModuleSpec factory functions in importlib.util are intended to be helpful for converting existing finders. from_loader() and from_file_location() are both straight-forward utilities in this regard. In the case where loaders already expose methods for creating and preparing modules, ModuleSpec.from_module() may be useful to the corresponding finder. For existing loaders, exec_module() should be a relatively direct conversion from the non-boilerplate portion of load_module(). In some uncommon cases the loader should also implement create_module(). ModuleSpec Users ================ ModuleSpec objects have 3 distinct target audiences: Python itself, import hooks, and normal Python users. Python will use specs in the import machinery, in interpreter startup, and in various standard library modules. Some modules are import-oriented, like pkgutil, and others are not, like pickle and pydoc. In all cases, the full ModuleSpec API will get used. Import hooks (finders and loaders) will make use of the spec in specific ways. First of all, finders may use the spec factory functions in importlib.util to create spec objects. They may also directly adjust the spec attributes after the spec is created. Secondly, the finder may bind additional information to the spec (in finder_extras) for the loader to consume during module creation/execution. Finally, loaders will make use of the attributes on a spec when creating and/or executing a module. Python users will be able to inspect a module's ``__spec__`` to get import-related information about the object. Generally, Python applications and interactive users will not be using the ``ModuleSpec`` factory functions nor any the instance methods. How Loading Will Work ===================== This is an outline of what happens in ModuleSpec's loading functionality:: def load(spec): if not hasattr(spec.loader, 'exec_module'): module = spec.loader.load_module(spec.name) spec.init_module_attrs(module) return sys.modules[spec.name] module = None if hasattr(spec.loader, 'create_module'): module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) spec.init_module_attrs(module) sys.modues[spec.name] = module try: spec.loader.exec_module(module) except BaseException: try: del sys.modules[spec.name] except KeyError: pass raise return sys.modules[spec.name] These steps are exactly what Loader.load_module() is already expected to do. Loaders will thus be simplified since they will only need to implement exec_module(). Note that we must return the module from sys.modules. During loading the module may have replaced itself in sys.modules. Since we don't have a post-import hook API to accommodate the use case, we have to deal with it. However, in the replacement case we do not worry about setting the import-related module attributes on the object. The module writer is on their own if they are doing this. ModuleSpec ========== Attributes ---------- Each of the following names is an attribute on ModuleSpec objects. A value of None indicates "not set". This contrasts with module objects where the attribute simply doesn't exist. Most of the attributes correspond to the import-related attributes of modules. Here is the mapping. The reverse of this mapping is used by ModuleSpec.init_module_attrs(). ========================== ============== On ModuleSpec On Modules ========================== ============== name __name__ loader __loader__ package __package__ origin __file__* cached __cached__*,** submodule_search_locations __path__** loader_state \- has_location \- ========================== ============== | \* Set on the module only if spec.has_location is true. | \*\* Set on the module only if the spec attribute is not None. While package and has_location are read-only properties, the remaining attributes can be replaced after the module spec is created and even after import is complete. This allows for unusual cases where directly modifying the spec is the best option. However, typical use should not involve changing the state of a module's spec. **origin** "origin" is a string for the name of the place from which the module originates. See `origin`_ above. Aside from the informational value, it is also used in module_repr(). In the case of a spec where "has_location" is true, ``__file__`` is set to the value of "origin". For built-in modules "origin" would be set to "built-in". **has_location** As explained in the `location`_ section above, many modules are "locatable", meaning there is a corresponding resource from which the module will be loaded and that resource can be described by a string. In contrast, non-locatable modules can't be loaded in this fashion, e.g. builtin modules and modules dynamically created in code. For these, the name is the only way to access them, so they have an "origin" but not a "location". "has_location" is true if the module is locatable. In that case the spec's origin is used as the location and ``__file__`` is set to spec.origin. If additional location information is required (e.g. zipimport), that information may be stored in spec.loader_state. "has_location" may be implied from the existence of a load_data() method on the loader. Incidently, not all locatable modules will be cachable, but most will. **submodule_search_locations** The list of location strings, typically directory paths, in which to search for submodules. If the module is a package this will be set to a list (even an empty one). Otherwise it is None. The name of the corresponding module attribute, ``__path__``, is relatively ambiguous. Instead of mirroring it, we use a more explicit name that makes the purpose clear. **loader_state** A finder may set loader_state to any value to provide additional data for the loader to use during loading. A value of None is the default and indicates that there is no additional data. Otherwise it can be set to any object, such as a dict, list, or types.SimpleNamespace, containing the relevant extra information. For example, zipimporter could use it to pass the zip archive name to the loader directly, rather than needing to derive it from origin or create a custom loader for each find operation. loader_state is meant for use by the finder and corresponding loader. It is not guaranteed to be a stable resource for any other use. Factory Functions ----------------- **spec_from_file_location(name, location, \*, loader=None, submodule_search_locations=None)** Build a spec from file-oriented information and loader APIs. * "origin" will be set to the location. * "has_location" will be set to True. * "cached" will be set to the result of calling cache_from_source(). * "origin" can be deduced from loader.get_filename() (if "location" is not passed in. * "loader" can be deduced from suffix if the location is a filename. * "submodule_search_locations" can be deduced from loader.is_package() and from os.path.dirname(location) if locatin is a filename. **from_loader(name, loader, \*, origin=None, is_package=None)** Build a spec with missing information filled in by using loader APIs. * "has_location" can be deduced from loader.get_data. * "origin" can be deduced from loader.get_filename(). * "submodule_search_locations" can be deduced from loader.is_package() and from os.path.dirname(location) if locatin is a filename. **spec_from_module(module, loader=None)** Build a spec based on the import-related attributes of an existing module. The spec attributes are set to the corresponding import- related module attributes. See the table in `Attributes`_. Omitted Attributes and Methods ------------------------------ There is no "PathModuleSpec" subclass of ModuleSpec that separates out has_location, cached, and submodule_search_locations. While that might make the separation cleaner, module objects don't have that distinction. ModuleSpec will support both cases equally well. While "is_package" would be a simple additional attribute (aliasing self.submodule_search_locations is not None), it perpetuates the artificial (and mostly erroneous) distinction between modules and packages. Conceivably, a ModuleSpec.load() method could optionally take a list of modules with which to interact instead of sys.modules. That capability is left out of this PEP, but may be pursued separately at some other time, including relative to PEP 406 (import engine). Likewise load() could be leveraged to implement multi-version imports. While interesting, doing so is outside the scope of this proposal. Others: * Add ModuleSpec.submodules (RO-property) - returns possible submodules relative to the spec. * Add ModuleSpec.loaded (RO-property) - the module in sys.module, if any. * Add ModuleSpec.data - a descriptor that wraps the data API of the spec's loader. * Also see [cleaner_reload_support]_. Backward Compatibility ---------------------- ModuleSpec doesn't have any. This would be a different story if Finder.find_module() were to return a module spec instead of loader. In that case, specs would have to act like the loader that would have been returned instead. Doing so would be relatively simple, but is an unnecessary complication. It was part of earlier versions of this PEP. Subclassing ----------- Subclasses of ModuleSpec are allowed, but should not be necessary. Simply setting loader_state or adding functionality to a custom finder or loader will likely be a better fit and should be tried first. However, as long as a subclass still fulfills the requirements of the import system, objects of that type are completely fine as the return value of Finder.find_spec(). Existing Types ============== Module Objects -------------- Other than adding ``__spec__``, none of the import-related module attributes will be changed or deprecated, though some of them could be; any such deprecation can wait until Python 4. A module's spec will not be kept in sync with the corresponding import- related attributes. Though they may differ, in practice they will typically be the same. One notable exception is that case where a module is run as a script by using the ``-m`` flag. In that case ``module.__spec__.name`` will reflect the actual module name while ``module.__name__`` will be ``__main__``. A module's spec is not guaranteed to be identical between two modules with the same name. Likewise there is no guarantee that successive calls to importlib.find_spec() will return the same object or even an equivalent object, though at least the latter is likely. Finders ------- Finders are still responsible for creating the loader. That loader will now be stored in the module spec returned by find_spec() rather than returned directly. As is currently the case without the PEP, if a loader would be costly to create, that loader can be designed to defer the cost until later. **MetaPathFinder.find_spec(name, path=None)** **PathEntryFinder.find_spec(name)** Finders will return ModuleSpec objects when find_spec() is called. This new method replaces find_module() and find_loader() (in the PathEntryFinder case). If a loader does not have find_spec(), find_module() and find_loader() are used instead, for backward-compatibility. Adding yet another similar method to loaders is a case of practicality. find_module() could be changed to return specs instead of loaders. This is tempting because the import APIs have suffered enough, especially considering PathEntryFinder.find_loader() was just added in Python 3.3. However, the extra complexity and a less-than- explicit method name aren't worth it. Loaders ------- **Loader.exec_module(module)** Loaders will have a new method, exec_module(). Its only job is to "exec" the module and consequently populate the module's namespace. It is not responsible for creating or preparing the module object, nor for any cleanup afterward. It has no return value. exec_module() will be used during both loading and reloading. exec_module() should properly handle the case where it is called more than once. For some kinds of modules this may mean raising ImportError every time after the first time the method is called. This is particularly relevant for reloading, where some kinds of modules do not support in-place reloading. **Loader.create_module(spec)** Loaders may also implement create_module() that will return a new module to exec. It may return None to indicate that the default module creation code should be used. One use case, though atypical, for create_module() is to provide a module that is a subclass of the builtin module type. Most loaders will not need to implement create_module(), create_module() should properly handle the case where it is called more than once for the same spec/module. This may include returning None or raising ImportError. .. note:: exec_module() and create_module() should not set any import-related module attributes. The fact that load_module() does is a design flaw that this proposal aims to correct. Other changes: PEP 420 introduced the optional module_repr() loader method to limit the amount of special-casing in the module type's ``__repr__()``. Since this method is part of ModuleSpec, it will be deprecated on loaders. However, if it exists on a loader it will be used exclusively. Loader.init_module_attr() method, added prior to Python 3.4's release , will be removed in favor of the same method on ModuleSpec. However, InspectLoader.is_package() will not be deprecated even though the same information is found on ModuleSpec. ModuleSpec can use it to populate its own is_package if that information is not otherwise available. Still, it will be made optional. One consequence of ModuleSpec is that loader ``__init__`` methods will no longer need to accommodate per-module state. The path-based loaders in importlib take arguments in their ``__init__()`` and have corresponding attributes. However, the need for those values is eliminated by module specs. In addition to executing a module during loading, loaders will still be directly responsible for providing APIs concerning module-related data. Other Changes ============= * The various finders and loaders provided by importlib will be updated to comply with this proposal. * The spec for the ``__main__`` module will reflect how the interpreter was started. For instance, with ``-m`` the spec's name will be that of the run module, while ``__main__.__name__`` will still be "__main__". * We add importlib.find_spec() to mirror importlib.find_loader() (which becomes deprecated). * importlib.reload() is changed to use ModuleSpec.load(). * importlib.reload() will now make use of the per-module import lock. Reference Implementation ======================== A reference implementation will be available at http://bugs.python.org/issue18864. Open Issues ============== \* The impact of this change on pkgutil (and setuptools) needs looking into. It has some generic function-based extensions to PEP 302. These may break if importlib starts wrapping loaders without the tools' knowledge. \* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc, inspect. For instance, pickle should be updated in the ``__main__`` case to look at ``module.__spec__.name``. \* Impact on some kinds of lazy loading modules. [lazy_import_concerns]_ References ========== .. [ref_files_pep] http://mail.python.org/pipermail/import-sig/2013-August/000658.html .. [import_system_docs] http://docs.python.org/3/reference/import.html .. [cleaner_reload_support] https://mail.python.org/pipermail/import-sig/2013-September/000735.html .. [lazy_import_concerns] https://mail.python.org/pipermail/python-dev/2013-August/128129.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

3 5

PEP 454 (tracemalloc): new minimalist version
by Victor Stinner 20 Oct '13

20 Oct '13

Hi, I plan to push the new tracemalloc module this week-end, before the next (and last) alpha version, except if someone complains :-) I prefer to have one month before the first beta to have more time to test the module. So if a major issue is raised, we may remove the tracemalloc module before the final version. Charles-François Natali reviewed my PEP 454 (tracemalloc). He suggested me to split the module into two parts: keep only essential parts in the tracemalloc Python (added to Python stdlib), and move other tools to an external tool (hosted on PyPI). We may integrate more functions (especially metrics) later (in Python 3.5), after they are well tested externally. I updated the PEP and the implementation. I will update the patch on the bug tracker later. HTML version of the PEP: http://www.python.org/dev/peps/pep-0454/ Since the previous version of the PEP, almost all changes are removed parts: * Remove tasks: the tool now uses classic threads * Remove functions to get metrics: the tool is now responsible to add them to a snapshot * Remove DisplayTop and DisplayTopTask class * Remove TakeSnaphot class PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations record the C filename and line number where the allocation occurs. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These tools analyze objects type and/or content. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Computed differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === Main Functions -------------- ``clear_traces()`` function: Clear traces and statistics on Python memory allocations, and reset the ``get_traced_memory()`` counter. ``disable()`` function: Stop tracing Python memory allocations. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. At fork, the module is automatically disabled in the child process. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module as a tuple: ``(size: int, free: int)``. * *size*: total size of bytes allocated by the module, including *free* bytes * *free*: number of free bytes available to store data ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. Trace Functions --------------- ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace of a memory block. Use the ``set_traceback_limit()`` function to change the limit. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. See also ``get_object_trace()`` and ``gc.get_referrers()`` functions. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. The function only returns the trace of the main memory block of the object. The *size* of the trace is smaller than the total size of the object if the object is composed by more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``get_trace()``, ``get_traces()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_trace()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traces()`` function: Get traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``(size: int, traceback)`` and *traceback* is a list of ``(filename: str, lineno: int)``. *traceback* can be empty, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_trace()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace of a memory block. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``clear_traces()`` function to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filter matchs the filename and line number, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filter matchs the filename, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the filter matchs the line number, ``False`` otherwise. ``match_traceback(traceback)`` method: Return ``True`` if the filter matchs the *traceback*, ``False`` otherwise. *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or less than ``1``, it matches any line number. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class: Top of allocated memory blocks grouped by *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. The ``StatsDiff.differences`` list is not sorted: call the ``StatsDiff.sort()`` method to sort the list. ``None`` values are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``group_by`` attribute: Determine how memory allocations were grouped: see ``Snapshot.top_by()`` for the available values. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* is a ``(size: int, count: int)`` tuple. See the ``Snapshot.top_by()`` method. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Metric ------ ``Metric(name: str, value: int, format: str)`` class: Value of a metric when a snapshot is created. ``name`` attribute: Name of the metric. ``value`` attribute: Value of the metric. ``format`` attribute: Format of the metric (``str``). Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traces: dict=None, stats: dict=None)`` class: Snapshot of traces and statistics on memory blocks allocated by Python. ``add_metric(name: str, value: int, format: str)`` method: Helper to add a ``Metric`` instance to ``Snapshot.metrics``. Return the newly created ``Metric`` instance. Raise an exception if the name is already present in ``Snapshot.metrics``. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot. See the the ``enable()`` function. ``get_metric(name, default=None)`` method: Get the value of the metric called *name*. Return *default* if the metric does not exist. ``load(filename, traces=True)`` classmethod: Load a snapshot from a file. If *traces* is ``False``, don't load traces. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ================================ group_by description key type ===================== ======================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` traceback ``(address: int, traceback)`` ===================== ======================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is less than ``2``. ``write(filename)`` method: Write the snapshot into a file. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in a trace of a memory block allocated by Python. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. StatsDiff --------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest difference to the smallest difference. Sort by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the ``Snapshot.top_by()`` method. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed in the public domain. Victor

5 9

An interesting exception handling quirk
by Nick Coghlan 20 Oct '13

20 Oct '13

I've been looking at Terry's suggestion of using a class based implementation for contextlib.suppress, and rediscovered why I suggested Raymond switch the original implementation to using @contextmanager: recreating the *exact* exception subclass check from Python is actually difficult these days. The exception handling only looks for concrete subclasses, ignoring the dynamic subclass hook: >>> import abc >>> class ExceptionABC(abc.ABC, Exception): ... pass ... >>> ExceptionABC.register(ZeroDivisionError) <class 'ZeroDivisionError'> >>> issubclass(ZeroDivisionError, ExceptionABC) True >>> try: ... 1/0 ... except ExceptionABC: ... print("It's a subclass of ExceptionABC!") ... except ZeroDivisionError: ... print("But not according to the exception handling!") ... But not according to the exception handling! I don't actually have a problem with the current behaviour (since raise and except both enforce a requirement for the types involved to be concrete subclasses of BaseException), but as far as I'm aware we don't explicitly state anywhere that if exception handling and an issubclass check give different answers for whether or not exception A is a subclass of exception B, then the fault for any resulting differences in behavious lies with the way A and/or B are defined, not with the code using issubclass to check for exception inheritance. Having noticed the discrepancy, I feel like it should be explicitly recorded somewhere in the language reference, I'm just not sure where. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 4

Recall: cpython: Rename contextlib.ignored() to contextlib.ignore().
by Mark.Favas＠csiro.au 20 Oct '13

20 Oct '13

Favas, Mark (CSIRO IM&T, Kensington) would like to recall the message, "cpython: Rename contextlib.ignored() to contextlib.ignore().".

1 0

cpython: Rename contextlib.ignored() to contextlib.ignore().
by Mark.Favas＠csiro.au 20 Oct '13

20 Oct '13

Or perhaps call it "contextlib.ignore_first(Exception)"? Mark Favas iVEC@CSIRO Director A/Director, Technical Operations, iVEC Advanced Scientific Computing CSIRO IM&T

1 0

I *might* push Python 3.4.0 alpha 4 back a week
by Larry Hastings 20 Oct '13

20 Oct '13

A lot has landed in trunk in the last day or two: Tulip, Argument Clinic, and statistics just landed too. The buildbots are upset at us humans--there's a lot of red. See for yourself: http://buildbot.python.org/all/waterfall?category=3.x.stable I'd like to tag Alpha 4 late tonight, but if the buildbots are still mostly-red it's a bad idea. If you can help make the buildbots happy, and have some time in the next eight or ten hours, please hop in to #python-dev and ask what you can do to help. SOS -- Save Our Schedule, //arry/

2 1

Python-Dev October 2013