Mailman 3 September 2013 - Python-Dev

Re: [Python-Dev] cpython: Issue #18571: Implementation of the PEP 446: file descriptors and file handles
by Georg Brandl Sept. 15, 2013

Sept. 15, 2013

On 08/28/2013 01:20 AM, victor.stinner wrote: > http://hg.python.org/cpython/rev/ef889c3d5dc6 > changeset: 85420:ef889c3d5dc6 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Wed Aug 28 00:53:59 2013 +0200 > summary: > Issue #18571: Implementation of the PEP 446: file descriptors and file handles > are now created non-inheritable; add functions os.get/set_inheritable(), > os.get/set_handle_inheritable() and socket.socket.get/… [View More]

2 1

Re: [Python-Dev] [Python-checkins] cpython: Issue #18937: Add an assertLogs() context manager to unittest.TestCase to
by Terry Reedy Sept. 14, 2013

Sept. 14, 2013

On 9/14/2013 1:45 PM, antoine.pitrou wrote: > http://hg.python.org/cpython/rev/4f5815747f58 > changeset: 85701:4f5815747f58 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Sat Sep 14 19:45:47 2013 +0200 > summary: > Issue #18937: Add an assertLogs() context manager to unittest.TestCase to ensure that a block of code emits a message using the logging module. > > files: > Doc/library/unittest.rst | 41 +++++++ > Lib/… [View More]unittest/case.py | 109 +++++++++++++++++++- > Lib/unittest/test/test_case.py | 96 ++++++++++++++++++ > Misc/NEWS | 2 + > 4 files changed, 242 insertions(+), 6 deletions(-) > > > diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst > --- a/Doc/library/unittest.rst > +++ b/Doc/library/unittest.rst > @@ -1031,6 +1031,47 @@ > .. versionchanged:: 3.3 > Added the *msg* keyword argument when used as a context manager. assertWarnsRegex > + .. method:: assertLogs(logger=None, level=None) > + > + A context manager to test that at least one message is logged on > + the *logger* or one of its children, with at least the given > + *level*. > + > + If given, *logger* should be a :class:`logging.Logger` object or a > + :class:`str` giving the name of a logger. The default is the root > + logger, which will catch all messages. > + > + If given, *level* should be either a numeric logging level or > + its string equivalent (for example either ``"ERROR"`` or > + :attr:`logging.ERROR`). The default is :attr:`logging.INFO`. > + > + The test passes if at least one message emitted inside the ``with`` > + block matches the *logger* and *level* conditions, otherwise it fails. > + > + The object returned by the context manager is a recording helper > + which keeps tracks of the matching log messages. It has two > + attributes: > + > + .. attribute:: records > + > + A list of :class:`logging.LogRecord` objects of the matching > + log messages. > + > + .. attribute:: output > + > + A list of :class:`str` objects with the formatted output of > + matching messages. > + > + Example:: > + > + with self.assertLogs('foo', level='INFO') as cm: > + logging.getLogger('foo').info('first message') > + logging.getLogger('foo.bar').error('second message') > + self.assertEqual(cm.output, ['INFO:foo:first message', > + 'ERROR:foo.bar:second message']) > + > + .. versionadded:: 3.4 > + > > There are also other methods used to perform more specific checks, such as: The new method should be added to the table of raises/warns methods that starts this group. The table intro sentence "It is also possible to check that exceptions and warnings are raised using the following methods:" should be modified to something like "It is also possible to check the production of exception, warning, and log messages using the following methods:" Terry [View Less]

1 0

Summary of Python tracker Issues
by Python tracker Sept. 13, 2013

Sept. 13, 2013

ACTIVITY SUMMARY (2013-09-06 - 2013-09-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4212 (+17) closed 26571 (+48) total 30783 (+65) Open issues with patches: 1935 Issues opened (46) ================== #18939: Venv docs regarding original python install http://bugs.python.org/issue18939 reopened by gwideman #18948: deliberately crashing … [View More]

1 0

Add a "transformdict" to collections
by Antoine Pitrou Sept. 12, 2013

Sept. 12, 2013

Hello, In http://bugs.python.org/issue18986 I proposed adding a new mapping type to the collections module. The original use case is quite common in network programming and elsewhere (Eric Snow on the tracker mentioned an application with stock symbols). You want to have an associative container which matches keys case-insensitively but also preserves the original casing (e.g. for presentation). It is a commonly reimplemented container. It is also an instance of a more general pattern: match … [View More]

29 86

f_lineno accessors
by Xavier de Gaye Sept. 12, 2013

Sept. 12, 2013

In issues 7238 [1], 16482 [2], 17697 [3] and 17277 [4], the line number may be incorrect when the global trace function has been removed but not the frame f_trace function. A simple test (see below) in issue 17288 [5] crashes the interpreter when setting f_lineno in a generator from a return trace function. All those issues are few months old. There is a patch at issue 17277 [4] that fixes the first 4 issues. There is also a patch for issue 17288 [5]. ###### Setting f_lineno in a generator … [View More]

1 0

Need testing audio files
by Serhiy Storchaka Sept. 11, 2013

Sept. 11, 2013

I work on enhancement of audio modules testing [1], and I need free (in both senses) small sample audio files in different formats. We already have audiotest.au (mono, and sunau has a bug in processing multichannel files [2]) and Sine-1000Hz-300ms.aif, but this is not enough. I have generated a pack of files like Sine-1000Hz-300ms.aif by Python, it is enough for regression testing but only if current implementation is correct. I found some collections of sample files at [3], but I'm not … [View More]

8 11

importance of dir
by Ethan Furman Sept. 11, 2013

Sept. 11, 2013

http://docs.python.org/3/library/functions.html#dir: > > Note: Because dir() is supplied primarily as a convenience for > use at an interactive prompt [...] I suspect this comment is out of date, as there are two functions in the inspect module that rely on dir(), which also means that help indirectly relies on dir(). And we also now have the __dir__ method that we can use to customize what gets returned. > [...] more than it tries to supply a rigorously or consistently > … [View More]

2 1

IOCP (I/O Completion Port) in Python ?
by 张佩佩 Sept. 10, 2013

Sept. 10, 2013

Hello: I wondering why there is no standard IOCP module in Python ? As I know: Python3 have support epoll in linux and kqueue in freebsd. Is there a plan to add IOCP module for Windows ? Regards! peipei

3 2

Offtopic: OpenID Providers
by Jesus Cea Sept. 10, 2013

Sept. 10, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just received an email from my OpenID provider, "myOpenID", saying that they drop OpenID service next February. I wonder what other OpenID providers are used by other python-dev fellows. What are you using?. bugs.python.org admins could share some data? I agree than OpenID is (quite) dead, but I rather prefer OpenID to use user/pass. I have big hopes for Mozilla Persona, looking forward Python infrastructure support :). PS: I use "http://www.… [View More]

21 64

[RELEASED] Python 3.4.0a2
by Larry Hastings Sept. 10, 2013

Sept. 10, 2013

On behalf of the Python development team, I'm chuffed to announce the second alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding … [View More]

7 6

Re: [Python-Dev] PEP 447: add type.__locallookup__
by Armin Rigo Sept. 10, 2013

Sept. 10, 2013

Hi Mark, On Mon, Sep 9, 2013 at 11:18 PM, Mark Shannon <mark(a)hotpy.org> wrote: > 5. Other implementations. What do the Jython/IronPython/PyPy developers > think? Thanks for asking :-) I'm fine with staying out of language design issues like this one, and I believe it's the general concensus in PyPy. Whatever gets decided will probably be easy to port to PyPy and have no measurable performance impact: the extra checks on the type, if any, are all constant-folded by the JIT. A bientôt, Armin.

1 0

Re: [Python-Dev] cpython (2.7): Clarify mmap.close method behavior. Addresses issue #18815
by Antoine Pitrou Sept. 10, 2013

Sept. 10, 2013

On Tue, 10 Sep 2013 07:40:21 +0200 (CEST) senthil.kumaran <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/443d12b61e5b > changeset: 85653:443d12b61e5b > branch: 2.7 > parent: 85639:740bd510a888 > user: Senthil Kumaran <senthil(a)uthcode.com> > date: Mon Sep 09 22:38:58 2013 -0700 > summary: > Clarify mmap.close method behavior. Addresses issue #18815 > Patch contributed by Anoop Thomas Mathew. > &… [View More]

1 0

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
by Victor Stinner Sept. 9, 2013

Sept. 9, 2013

2013/9/9 Antoine Pitrou <solipsis(a)pitrou.net>: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner(a)gmail.com> a écrit : >> 2013/9/9 Larry Hastings <larry(a)hastings.org>: >> > Python 3.4 includes a range of improvements of the 3.x series, >> > including hundreds of small improvements and bug fixes. Major new >> > features and changes in the 3.4 release series so far include: >> > >> > * PEP 446, … [View More]

1 0

Re: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release fixups.
by Nick Coghlan Sept. 9, 2013

Sept. 9, 2013

On 9 Sep 2013 22:15, "larry.hastings" <python-checkins(a)python.org> wrote: > > http://hg.python.org/cpython/rev/6b211a0c8042 > changeset: 85645:6b211a0c8042 > user: Larry Hastings <larry(a)hastings.org> > date: Mon Sep 09 21:08:52 2013 +0900 > summary: > Post-3.4.0a2-release fixups. > > files: > Include/patchlevel.h | 2 +- > Misc/NEWS | 14 +++++++++++++- > 2 files changed, 14 insertions(+), 2 deletions(-) … [View More]

2 2

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
by R. David Murray Sept. 9, 2013

Sept. 9, 2013

On Mon, 09 Sep 2013 14:45:51 +0200, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner(a)gmail.com> a écrit : > > 2013/9/9 Larry Hastings <larry(a)hastings.org>: > > > Python 3.4 includes a range of improvements of the 3.x series, > > > including hundreds of small improvements and bug fixes. Major new > > > features and changes in the 3.4 release series so far include: > … [View More]

1 0

unicodedata module is out of date
by Andrew Miller Sept. 9, 2013

Sept. 9, 2013

The unicodedata module only contains data up to Unicode 5.2 (October 2009), so attempting to reference any character from a later version e.g: unicodedata.lookup("TURKISH LIRA SIGN") results in a KeyError. Also, it seems to be limited to properties in the UnicodeData.txt file and does not contain any data from the other files from the Unicode Character Database (the perl library Unicode::UCD is far more complete). Are there any plans to update this module to the latest Unicode version (6.2, … [View More]

7 10

DTRACE support
by Jesus Cea Sept. 9, 2013

Sept. 9, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. Python is embarrasingly missing from this list. Some examples: <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part…> <http://www.phpdeveloper.org/news/18859> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> I have spend a very long time on a patch for Dtrace support in most platforms with dtrace available. Currently working under … [View More]

9 19

RFC: PEP 454: Add a new tracemalloc module
by Victor Stinner Sept. 8, 2013

Sept. 8, 2013

Hi, Antoine Pitrou suggested me to write a PEP to discuss the API of the new tracemalloc module that I proposed to add to Python 3.4. Here you have. If you prefer to read the HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the current implementation of the module. http://hg.python.org/features/tracemalloc/file/tip/Doc/library/tracemalloc.… The documentaion contains examples and a short "tutorial". PEP: 454 Title: Add a new tracemalloc module to trace … [View More]Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace Python memory allocations. Rationale ========= Common debug tools tracing memory allocations read the C filename and number. Using such tool to analyze Python memory allocations does not help because most memory allocations are done in the same C function, ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python languages like ``Heapy`` and ``PySizer``. These projects analyze objects type and/or content. These tools are useful when the most memory leak are instances of the same type and this type in allocated only in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are allocated. Finding reference cycles is also a difficult task. There are different tools to draw a diagram of all references. These tools cannot be used huge on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. The hook can inspect the current Python frame to get the Python filename and line number. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory allocations made by Python. The module provides the following information: * Statistics on Python memory allocations per Python filename and line number: size, number, and average size of allocations * Compute differences between two snapshots of Python memory allocations * Location of a Python memory allocation: size in bytes, Python filename and line number Command line options ==================== The ``python -m tracemalloc`` command can be used to analyze and compare snapshots. The command takes a list of snapshot filenames and has the following options. ``-g``, ``--group-per-file`` Group allocations per filename, instead of grouping per line number. ``-n NTRACES``, ``--number NTRACES`` Number of traces displayed per top (default: 10). ``--first`` Compare with the first snapshot, instead of comparing with the previous snapshot. ``--include PATTERN`` Only include filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``--exclude PATTERN`` Exclude filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``-S``, ``--hide-size`` Hide the size of allocations. ``-C``, ``--hide-count`` Hide the number of allocations. ``-A``, ``--hide-average`` Hide the average size of allocations. ``-P PARTS``, ``--filename-parts=PARTS`` Number of displayed filename parts (default: 3). ``--color`` Force usage of colors even if ``sys.stdout`` is not a TTY device. ``--no-color`` Disable colors if ``sys.stdout`` is a TTY device. API === To trace the most Python memory allocations, the module should be enabled as early as possible in your application by calling ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. Functions --------- ``enable()`` function: Start tracing Python memory allocations. ``disable()`` function: Stop tracing Python memory allocations and stop the timer started by ``start_timer()``. ``is_enabled()`` function: Get the status of the module: ``True`` if it is enabled, ``False`` otherwise. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``trace`` instance. Return ``None`` if the tracemalloc module did not save the location when the object was allocated, for example if the module was disabled. ``get_process_memory()`` function: Get the memory usage of the current process as a meminfo namedtuple with two attributes: * ``rss``: Resident Set Size in bytes * ``vms``: size of the virtual memory in bytes Return ``None`` if the platform is not supported. Use the ``psutil`` module if available. ``get_stats()`` function: Get statistics on Python memory allocations per Python filename and per Python line number. Return a dictionary ``{filename: str -> {line_number: int -> stats: line_stat}}`` where *stats* in a ``line_stat`` instance. *filename* and *line_number* can be ``None``. Return an empty dictionary if the tracemalloc module is disabled. ``get_traces(obj)`` function: Get all traces of a Python memory allocations. Return a dictionary ``{pointer: int -> trace}`` where *trace* is a ``trace`` instance. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: Start a timer calling ``func(*args, **kwargs)`` every *delay* seconds. The timer is based on the Python memory allocator, it is not real time. *func* is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. If ``start_timer()`` is called twice, previous parameters are replaced. The timer has a resolution of 1 second. ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to run regulary a task. ``stop_timer()`` function: Stop the timer started by ``start_timer()``. trace class ----------- ``trace`` class: This class represents debug information of an allocated memory block. ``size`` attribute: Size in bytes of the memory block. ``filename`` attribute: Name of the Python script where the memory block was allocated, ``None`` if unknown. ``lineno`` attribute: Line number where the memory block was allocated, ``None`` if unknown. line_stat class ---------------- ``line_stat`` class: Statistics on Python memory allocations of a specific line number. ``size`` attribute: Total size in bytes of all memory blocks allocated on the line. ``count`` attribute: Number of memory blocks allocated on the line. DisplayTop class ---------------- ``DisplayTop(count: int=10, file=sys.stdout)`` class: Display the list of the *count* biggest memory allocations into *file*. ``display()`` method: Display the top once. ``start(delay: int)`` method: Start a task using ``tracemalloc`` timer to display the top every *delay* seconds. ``stop()`` method: Stop the task started by the ``DisplayTop.start()`` method ``color`` attribute: If ``True``, ``display()`` uses color. The default value is ``file.isatty()``. ``compare_with_previous`` attribute: If ``True`` (default value), ``display()`` compares with the previous snapshot. If ``False``, compare with the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``"..."``. ``group_per_file`` attribute: If ``True``, group memory allocations per Python filename. If ``False`` (default value), group allocation per Python line number. ``show_average`` attribute: If ``True`` (default value), ``display()`` shows the average size of allocations. ``show_count`` attribute: If ``True`` (default value), ``display()`` shows the number of allocations. ``show_size`` attribute: If ``True`` (default value), ``display()`` shows the size of allocations. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Snapshot class -------------- ``Snapshot()`` class: Snapshot of Python memory allocations. Use ``TakeSnapshot`` to take regulary snapshots. ``create(user_data_callback=None)`` method: Take a snapshot. If *user_data_callback* is specified, it must be a callable object returning a list of ``(title: str, format: str, value: int)``. *format* must be ``'size'``. The list must always have the same length and the same order to be able to compute differences between values. Example: ``[('Video memory', 'size', 234902)]``. ``filter_filenames(patterns: list, include: bool)`` method: Remove filenames not matching any pattern of *patterns* if *include* is ``True``, or remove filenames matching a pattern of *patterns* if *include* is ``False`` (exclude). See ``fnmatch.fnmatch()`` for the syntax of a pattern. ``load(filename)`` classmethod: Load a snapshot from a file. ``write(filename)`` method: Write the snapshot into a file. ``pid`` attribute: Identifier of the process which created the snapshot (int). ``process_memory`` attribute: Result of the ``get_process_memory()`` function, can be ``None``. ``stats`` attribute: Result of the ``get_stats()`` function (dict). ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``user_data`` attribute: Optional list of user data, result of *user_data_callback* in ``Snapshot.create()`` (default: None). TakeSnapshot class ------------------ ``TakeSnapshot`` class: Task taking snapshots of Python memory allocations: write them into files. By default, snapshots are written in the current directory. ``start(delay: int)`` method: Start a task taking a snapshot every delay seconds. ``stop()`` method: Stop the task started by the ``TakeSnapshot.start()`` method. ``take_snapshot()`` method: Take a snapshot. ``filename_template`` attribute: Template (``str``) used to create a filename. The following variables can be used in the template: * ``$pid``: identifier of the current process * ``$timestamp``: current date and time * ``$counter``: counter starting at 1 and incremented at each snapshot The default pattern is ``'tracemalloc-$counter.pickle'``. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Links ===== Python issues: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain. [View Less]

3 7

Re: [Python-Dev] cpython: Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC
by Antoine Pitrou Sept. 8, 2013

Sept. 8, 2013

On Sun, 8 Sep 2013 11:54:00 +0200 (CEST) victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/b7f6f6f59e91 > changeset: 85619:b7f6f6f59e91 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Sun Sep 08 11:53:09 2013 +0200 > summary: > Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC > [...] > > + if fcntl: > + def test_get_inheritable_cloexec(self): … [View More]

2 1

metaclasses, classes, instances, and proper nomenclature
by Ethan Furman Sept. 8, 2013

Sept. 8, 2013

I've run across two different ways to think about this: 1) the type of the first argument 2) where the method/attribute lives Since attributes don't take a first argument they default to 2: an instance attribute lives in the instance, a class attribute lives in the class, and a metaclass attribute lives in the metaclass. Methods, on the other hand, do take a first argument: an instance method takes itself, a class method takes the class, and a metaclass method takes the metaclass. … [View More]

2 1

Pre-PEP: Redesigning extension modules
by Stefan Behnel Sept. 8, 2013

Sept. 8, 2013

Hi, this has been subject to a couple of threads on python-dev already, for example: http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986 http://thread.gmane.org/gmane.comp.python.devel/141037/focus=141046 It originally came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Here's an initial attempt at a PEP for it. It is based on the (unfinished) ModuleSpec PEP, which is being discussed on the import-sig mailing list. … [View More]http://mail.python.org/pipermail/import-sig/2013-August/000688.html Stefan PEP: 4XX Title: Redesigning extension modules Version: $Revision$ Last-Modified: $Date$ Author: Stefan Behnel <stefan_ml at behnel.de> BDFL-Delegate: ??? Discussions-To: ??? Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2013 Python-Version: 3.4 Post-History: 23-Aug-2013 Resolution: Abstract ======== This PEP proposes a redesign of the way in which extension modules interact with the interpreter runtime. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve them by bringing extension modules closer to the way Python modules behave. An implication of this PEP is that extension modules can use arbitrary types for their module implementation and are no longer restricted to types.ModuleType. This makes it easy to support properties at the module level and to safely store arbitrary global state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Motivation ========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its fully qualified module name, FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it is not trivial to correctly add the module to sys.modules either. This is specifically a problem for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or reloading and it is neither easy nor efficient with the current infrastructure to support these features. This PEP also addresses these issues. The current process =================== Currently, extension modules export an initialisation function named "PyInit_modulename", named after the file name of the shared library. This function is executed by the import machinery and must return either NULL in the case of an exception, or a fully initialised module object. The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef struct. It then continues to initialise it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the FQMN of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The main problem in this process is the missing support for passing state into the module init function, and for safely passing state through to the module creation code. The proposal ============ The current extension module initialisation will be deprecated in favour of a new initialisation scheme. Since the current scheme will continue to be available, existing code will continue to work unchanged, including binary compatibility. Extension modules that support the new initialisation scheme must export a new public symbol "PyModuleCreate_modulename", where "modulename" is the name of the shared library. This mimics the previous naming convention for the "PyInit_modulename" function. This symbol must resolve to a C function with the following signature:: PyObject* (*PyModuleTypeCreateFunction)(PyObject* module_spec) The "module_spec" argument receives a "ModuleSpec" instance, as defined in PEP 4XX (FIXME). (All names are obviously up for debate and bike-shedding at this point.) When called, this function must create and return a type object, either a Python class or an extension type that is allocated on the heap. This type will be instantiated as module instance by the importer. There is no requirement for this type to be exactly or a subtype of types.ModuleType. Any type can be returned. This follows the current support for allowing arbitrary objects in sys.modules and makes it easier for extension modules to define a type that exactly matches their needs for holding module state. The constructor of this type must have the following signature:: def __init__(self, module_spec): The "module_spec" argument receives the same object as the one passed into the module type creation function. Implementation ============== XXX - not started Reloading and Sub-Interpreters ============================== To "reload" an extension module, the module create function is executed again and returns a new module type. This type is then instantiated as by the original module loader and replaces the previous entry in sys.modules. Once the last references to the previous module and its type are gone, both will be subject to normal garbage collection. Sub-interpreter support is an inherent property of the design. During import in the sub-interpreter, the module create function is executed and returns a new module type that is local to the sub-interpreter. Both the type and its module instance are subject to garbage collection in the sub-interpreter. Open questions ============== It is not immediately obvious how extensions should be handled that want to register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case (or could construct them from the module spec of the current module), although not the import file path. A C-API could be provided to register new module types in the current interpreter, given a user provided ModuleSpec. There is no inherent requirement for the module creation function to actually return a type. It could return a arbitrary callable that creates a 'modulish' object when called. Should there be a type check in place that makes sure that what it returns is a type? I don't currently see a need for this. Copyright ========= This document has been placed in the public domain. [View Less]

7 46

when to fix cross-version bugs?
by Ethan Furman Sept. 7, 2013

Sept. 7, 2013

I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values (otherwise, in subclasses, the str() was used instead). Should this be fixed in 3.3 as well? What guidelines determine when a bug is fixed in previous versions? -- ~Ethan~

4 4

inspect and metaclasses
by Ethan Furman Sept. 6, 2013

Sept. 6, 2013

Part of the fix for issue #18693 is to fix inspect to look in the metaclass for class attributes (http://bugs.python.org/issue18929). In inspect.py in function get_mro() we can either add the metaclass unconditionally, or only if it is not 'type'. If we add unconditionally, then help() adds the following: class A(builtins.object) | Hello and goodbye + | + | Method resolution order: + | A + | builtins.object + | builtins.type |… [View More]

4 8

Details of "import lock" in 3.3
by Jesus Cea Sept. 6, 2013

Sept. 6, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With importlib and other recent changes, what are the current details of "import lock"?. That is, the lock/locks held when Python code does "import", specially in the case of multithreading. Is that documented anywhere? Thanks! - -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ … [View More]

2 3

Summary of Python tracker Issues
by Python tracker Sept. 6, 2013

Sept. 6, 2013

ACTIVITY SUMMARY (2013-08-30 - 2013-09-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4195 (+13) closed 26523 (+47) total 30718 (+60) Open issues with patches: 1915 Issues opened (43) ================== #16826: Don't check for PYTHONCASEOK if interpreter started with -E http://bugs.python.org/issue16826 reopened by meador.inge #18709: SSL … [View More]

1 0