Mailman 3 September 2013 - Python-Dev

Re: [Python-Dev] PEP 447: add type.__locallookup__
by Armin Rigo 10 Sep '13

10 Sep '13

Hi Mark, On Mon, Sep 9, 2013 at 11:18 PM, Mark Shannon <mark(a)hotpy.org> wrote: > 5. Other implementations. What do the Jython/IronPython/PyPy developers > think? Thanks for asking :-) I'm fine with staying out of language design issues like this one, and I believe it's the general concensus in PyPy. Whatever gets decided will probably be easy to port to PyPy and have no measurable performance impact: the extra checks on the type, if any, are all constant-folded by the JIT. A bientôt, Armin.

1 0

Re: [Python-Dev] cpython (2.7): Clarify mmap.close method behavior. Addresses issue #18815
by Antoine Pitrou 10 Sep '13

10 Sep '13

On Tue, 10 Sep 2013 07:40:21 +0200 (CEST) senthil.kumaran <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/443d12b61e5b > changeset: 85653:443d12b61e5b > branch: 2.7 > parent: 85639:740bd510a888 > user: Senthil Kumaran <senthil(a)uthcode.com> > date: Mon Sep 09 22:38:58 2013 -0700 > summary: > Clarify mmap.close method behavior. Addresses issue #18815 > Patch contributed by Anoop Thomas Mathew. > > files: > Doc/library/mmap.rst | 5 +++-- > 1 files changed, 3 insertions(+), 2 deletions(-) > > > diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst > --- a/Doc/library/mmap.rst > +++ b/Doc/library/mmap.rst > @@ -152,8 +152,9 @@ > > .. method:: close() > > - Close the file. Subsequent calls to other methods of the object will > - result in an exception being raised. > + Closes the mmap. Subsequent calls to other methods of the object will > + result in a ValueError exception being raised. This will not close > + the open file. I think this could be improved. Anonymous mmap objects don't have an "open file". Also, it would be clearer if it read "the underlying file" rather than "the underlying file object" rather than "the open file", because an mmap object is a file as well (it has read(), write(), seek(), etc.). Regards Antoine.

1 0

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
by Victor Stinner 09 Sep '13

09 Sep '13

2013/9/9 Antoine Pitrou <solipsis(a)pitrou.net>: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner(a)gmail.com> a écrit : >> 2013/9/9 Larry Hastings <larry(a)hastings.org>: >> > Python 3.4 includes a range of improvements of the 3.x series, >> > including hundreds of small improvements and bug fixes. Major new >> > features and changes in the 3.4 release series so far include: >> > >> > * PEP 446, changing file descriptors to not be inherited by default >> > in subprocesses >> >> The title of the PEP is "Make newly created file descriptors >> non-inheritable". It has an impact on all functions creating files and >> sockets not only the subprocess module. > > I don't think Larry's description is wrong. "Non-inheritable" is a > shorthand for "non-inheritable in subprocesses" with "subprocesses" > taken in the general sense (i.e. not only created with the subprocess > module). Oh, I misunderstood "in subprocesses", I read "in the subprocess module". The definition of FD inheritance is tricky. For example, on UNIX "non-inheritable" file descriptors are still inherited at fork :-) I hope that the documentation is explicit enough: http://docs.python.org/dev/library/os.html#inheritance-of-file-descriptors Victor

1 0

Re: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release fixups.
by Nick Coghlan 09 Sep '13

09 Sep '13

On 9 Sep 2013 22:15, "larry.hastings" <python-checkins(a)python.org> wrote: > > http://hg.python.org/cpython/rev/6b211a0c8042 > changeset: 85645:6b211a0c8042 > user: Larry Hastings <larry(a)hastings.org> > date: Mon Sep 09 21:08:52 2013 +0900 > summary: > Post-3.4.0a2-release fixups. > > files: > Include/patchlevel.h | 2 +- > Misc/NEWS | 14 +++++++++++++- > 2 files changed, 14 insertions(+), 2 deletions(-) > > > diff --git a/Include/patchlevel.h b/Include/patchlevel.h > --- a/Include/patchlevel.h > +++ b/Include/patchlevel.h > @@ -23,7 +23,7 @@ > #define PY_RELEASE_SERIAL 2 > > /* Version as a string */ > -#define PY_VERSION "3.4.0a2" > +#define PY_VERSION "3.4.0a2+" > /*--end constants--*/ > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -2,10 +2,22 @@ > Python News > +++++++++++ > > +What's New in Python 3.4.0 Alpha 3? > +=================================== > + > +Projected Release date: 2013-09-29 > + > +Core and Builtins > +----------------- > + > +Library > +------- > + > + I had already pushed alpha 3 entries in NEWS, so something seems to have gone wrong here. Perhaps, if RMs are preparing the release out of tree, we could get the NEWS file headings on default updated immediately after the last included commit? Cheers, Nick. > What's New in Python 3.4.0 Alpha 2? > =================================== > > -Projected Release date: 2013-09-08 > +Release date: 2013-09-09 > > Core and Builtins > ----------------- > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

2 2

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2
by R. David Murray 09 Sep '13

09 Sep '13

On Mon, 09 Sep 2013 14:45:51 +0200, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner(a)gmail.com> a écrit : > > 2013/9/9 Larry Hastings <larry(a)hastings.org>: > > > Python 3.4 includes a range of improvements of the 3.x series, > > > including hundreds of small improvements and bug fixes. Major new > > > features and changes in the 3.4 release series so far include: > > > > > > * PEP 446, changing file descriptors to not be inherited by default > > > in subprocesses > > > > The title of the PEP is "Make newly created file descriptors > > non-inheritable". It has an impact on all functions creating files and > > sockets not only the subprocess module. > > I don't think Larry's description is wrong. "Non-inheritable" is a > shorthand for "non-inheritable in subprocesses" with "subprocesses" > taken in the general sense (i.e. not only created with the subprocess > module). Not wrong, but definitely confusing. It is worth clarifying *somehow* that this does not apply only to the subprocess module, which is what a naive (or fast) reader will assume. --David

1 0

unicodedata module is out of date
by Andrew Miller 09 Sep '13

09 Sep '13

The unicodedata module only contains data up to Unicode 5.2 (October 2009), so attempting to reference any character from a later version e.g: unicodedata.lookup("TURKISH LIRA SIGN") results in a KeyError. Also, it seems to be limited to properties in the UnicodeData.txt file and does not contain any data from the other files from the Unicode Character Database (the perl library Unicode::UCD is far more complete). Are there any plans to update this module to the latest Unicode version (6.2, with 6.3 being released shortly), or is there another module that provides more up to date information? Thanks, Andrew

7 10

DTRACE support
by Jesus Cea 09 Sep '13

09 Sep '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. Python is embarrasingly missing from this list. Some examples: <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part…> <http://www.phpdeveloper.org/news/18859> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> I have spend a very long time on a patch for Dtrace support in most platforms with dtrace available. Currently working under Solaris and derivatives, and MacOS X. Last time I checked, it would crash FreeBSD because bugs in the dtrace port, but that was a long time ago. I would like to push this to Python 3.4, and the window is going to be closed soon, so I think this is the time to ask for opinions and support here. Does Python-Dev have any opinion or interest in this project?. Should I push for it? Some (not current) details: http://bugs.python.org/issue13405 DTrace is amazing: http://dtrace.org/blogs/ PS: I can't release this code as a PyPI project, because it mess with the inner core of Python. - -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUinv55lgi5GaxT1NAQIK7gP8DeaWxNVwNFa5PnfZe00Aay5l1rWzgj8d CY0D3W+PAdPkBci9SYPmfv7ajXrQWo/ccANYIRaUdI/U9Zjq/od7eNemOFqyL7U6 BrQpAUMySI6tMlL+gYEfQ8O47SManvTqoyNvOFAz9mVJute8IxKsbCIK/jiRHDXz vWyG7YrYN1A= =4E7+ -----END PGP SIGNATURE-----

9 19

RFC: PEP 454: Add a new tracemalloc module
by Victor Stinner 08 Sep '13

08 Sep '13

Hi, Antoine Pitrou suggested me to write a PEP to discuss the API of the new tracemalloc module that I proposed to add to Python 3.4. Here you have. If you prefer to read the HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the current implementation of the module. http://hg.python.org/features/tracemalloc/file/tip/Doc/library/tracemalloc.… The documentaion contains examples and a short "tutorial". PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace Python memory allocations. Rationale ========= Common debug tools tracing memory allocations read the C filename and number. Using such tool to analyze Python memory allocations does not help because most memory allocations are done in the same C function, ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python languages like ``Heapy`` and ``PySizer``. These projects analyze objects type and/or content. These tools are useful when the most memory leak are instances of the same type and this type in allocated only in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are allocated. Finding reference cycles is also a difficult task. There are different tools to draw a diagram of all references. These tools cannot be used huge on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. The hook can inspect the current Python frame to get the Python filename and line number. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory allocations made by Python. The module provides the following information: * Statistics on Python memory allocations per Python filename and line number: size, number, and average size of allocations * Compute differences between two snapshots of Python memory allocations * Location of a Python memory allocation: size in bytes, Python filename and line number Command line options ==================== The ``python -m tracemalloc`` command can be used to analyze and compare snapshots. The command takes a list of snapshot filenames and has the following options. ``-g``, ``--group-per-file`` Group allocations per filename, instead of grouping per line number. ``-n NTRACES``, ``--number NTRACES`` Number of traces displayed per top (default: 10). ``--first`` Compare with the first snapshot, instead of comparing with the previous snapshot. ``--include PATTERN`` Only include filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``--exclude PATTERN`` Exclude filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``-S``, ``--hide-size`` Hide the size of allocations. ``-C``, ``--hide-count`` Hide the number of allocations. ``-A``, ``--hide-average`` Hide the average size of allocations. ``-P PARTS``, ``--filename-parts=PARTS`` Number of displayed filename parts (default: 3). ``--color`` Force usage of colors even if ``sys.stdout`` is not a TTY device. ``--no-color`` Disable colors if ``sys.stdout`` is a TTY device. API === To trace the most Python memory allocations, the module should be enabled as early as possible in your application by calling ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. Functions --------- ``enable()`` function: Start tracing Python memory allocations. ``disable()`` function: Stop tracing Python memory allocations and stop the timer started by ``start_timer()``. ``is_enabled()`` function: Get the status of the module: ``True`` if it is enabled, ``False`` otherwise. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``trace`` instance. Return ``None`` if the tracemalloc module did not save the location when the object was allocated, for example if the module was disabled. ``get_process_memory()`` function: Get the memory usage of the current process as a meminfo namedtuple with two attributes: * ``rss``: Resident Set Size in bytes * ``vms``: size of the virtual memory in bytes Return ``None`` if the platform is not supported. Use the ``psutil`` module if available. ``get_stats()`` function: Get statistics on Python memory allocations per Python filename and per Python line number. Return a dictionary ``{filename: str -> {line_number: int -> stats: line_stat}}`` where *stats* in a ``line_stat`` instance. *filename* and *line_number* can be ``None``. Return an empty dictionary if the tracemalloc module is disabled. ``get_traces(obj)`` function: Get all traces of a Python memory allocations. Return a dictionary ``{pointer: int -> trace}`` where *trace* is a ``trace`` instance. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: Start a timer calling ``func(*args, **kwargs)`` every *delay* seconds. The timer is based on the Python memory allocator, it is not real time. *func* is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. If ``start_timer()`` is called twice, previous parameters are replaced. The timer has a resolution of 1 second. ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to run regulary a task. ``stop_timer()`` function: Stop the timer started by ``start_timer()``. trace class ----------- ``trace`` class: This class represents debug information of an allocated memory block. ``size`` attribute: Size in bytes of the memory block. ``filename`` attribute: Name of the Python script where the memory block was allocated, ``None`` if unknown. ``lineno`` attribute: Line number where the memory block was allocated, ``None`` if unknown. line_stat class ---------------- ``line_stat`` class: Statistics on Python memory allocations of a specific line number. ``size`` attribute: Total size in bytes of all memory blocks allocated on the line. ``count`` attribute: Number of memory blocks allocated on the line. DisplayTop class ---------------- ``DisplayTop(count: int=10, file=sys.stdout)`` class: Display the list of the *count* biggest memory allocations into *file*. ``display()`` method: Display the top once. ``start(delay: int)`` method: Start a task using ``tracemalloc`` timer to display the top every *delay* seconds. ``stop()`` method: Stop the task started by the ``DisplayTop.start()`` method ``color`` attribute: If ``True``, ``display()`` uses color. The default value is ``file.isatty()``. ``compare_with_previous`` attribute: If ``True`` (default value), ``display()`` compares with the previous snapshot. If ``False``, compare with the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``"..."``. ``group_per_file`` attribute: If ``True``, group memory allocations per Python filename. If ``False`` (default value), group allocation per Python line number. ``show_average`` attribute: If ``True`` (default value), ``display()`` shows the average size of allocations. ``show_count`` attribute: If ``True`` (default value), ``display()`` shows the number of allocations. ``show_size`` attribute: If ``True`` (default value), ``display()`` shows the size of allocations. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Snapshot class -------------- ``Snapshot()`` class: Snapshot of Python memory allocations. Use ``TakeSnapshot`` to take regulary snapshots. ``create(user_data_callback=None)`` method: Take a snapshot. If *user_data_callback* is specified, it must be a callable object returning a list of ``(title: str, format: str, value: int)``. *format* must be ``'size'``. The list must always have the same length and the same order to be able to compute differences between values. Example: ``[('Video memory', 'size', 234902)]``. ``filter_filenames(patterns: list, include: bool)`` method: Remove filenames not matching any pattern of *patterns* if *include* is ``True``, or remove filenames matching a pattern of *patterns* if *include* is ``False`` (exclude). See ``fnmatch.fnmatch()`` for the syntax of a pattern. ``load(filename)`` classmethod: Load a snapshot from a file. ``write(filename)`` method: Write the snapshot into a file. ``pid`` attribute: Identifier of the process which created the snapshot (int). ``process_memory`` attribute: Result of the ``get_process_memory()`` function, can be ``None``. ``stats`` attribute: Result of the ``get_stats()`` function (dict). ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``user_data`` attribute: Optional list of user data, result of *user_data_callback* in ``Snapshot.create()`` (default: None). TakeSnapshot class ------------------ ``TakeSnapshot`` class: Task taking snapshots of Python memory allocations: write them into files. By default, snapshots are written in the current directory. ``start(delay: int)`` method: Start a task taking a snapshot every delay seconds. ``stop()`` method: Stop the task started by the ``TakeSnapshot.start()`` method. ``take_snapshot()`` method: Take a snapshot. ``filename_template`` attribute: Template (``str``) used to create a filename. The following variables can be used in the template: * ``$pid``: identifier of the current process * ``$timestamp``: current date and time * ``$counter``: counter starting at 1 and incremented at each snapshot The default pattern is ``'tracemalloc-$counter.pickle'``. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Links ===== Python issues: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain.

3 7

Re: [Python-Dev] cpython: Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC
by Antoine Pitrou 08 Sep '13

08 Sep '13

On Sun, 8 Sep 2013 11:54:00 +0200 (CEST) victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/b7f6f6f59e91 > changeset: 85619:b7f6f6f59e91 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Sun Sep 08 11:53:09 2013 +0200 > summary: > Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC > [...] > > + if fcntl: > + def test_get_inheritable_cloexec(self): The right way to do this would be to skip the test if fcntl doesn't exist. Regards Antoine.

2 1

metaclasses, classes, instances, and proper nomenclature
by Ethan Furman 08 Sep '13

08 Sep '13

I've run across two different ways to think about this: 1) the type of the first argument 2) where the method/attribute lives Since attributes don't take a first argument they default to 2: an instance attribute lives in the instance, a class attribute lives in the class, and a metaclass attribute lives in the metaclass. Methods, on the other hand, do take a first argument: an instance method takes itself, a class method takes the class, and a metaclass method takes the metaclass. Going by option 1 above there is only one way to get an instance method, and only one way to get a metaclass method -- calling with the instance (either directly or indirectly via the class), or calling a metaclass method that has been marked as a @classmethod. Therein lies my confusion. class Meta(type): @classmethod def meta_method(mcls): print("I'm a metaclass method!") def cls_method1(cls): print("I'm a class method! Aren't I?") class Class(metaclass=Meta): @classmethod def cls_method2(cls): print("I'm a class method for sure!") def instance_method(self): print("And I'm a regular ol' instance method") So, is Meta.cls_method1 a class method? On the one hand, it takes the class as it's first parameter, on the other hand it lives in the metaclass. And on the third hand you can't get to it from the instance Class(). If you're wondering why this is posted to PyDev, the related question is this: What is the proper role of a metaclass? Should it basically fiddle with the class creation process and then get out of the way? The case in point is, of course, Enum. Currently it has a custom __getattr__, but it lives in the metaclass, EnumMeta. Now this is handy, because it means that Color.red.blue raises an AttributeError, where if __getattr__ lived in Enum itself that would work. It also has the __members__ attribute living in EnumMeta, which means it's not accessible from Color.red. In other words, EnumMeta is not getting out the way, it is still very much involved. On the one hand, that's cool; on the other hand, I hand to think hard to figure out why Color.red.blue was not getting routed through EnumMeta's __getattr__, but was instead getting routed through object.__getattr__. -- ~Ethan~

2 1