Mailman 3 October 2013 - Python-Dev

Support keyword in PEP URL?
by Victor Stinner 12 Oct '13

12 Oct '13

Hi, What do you think of adding an optional identifier to a PEP to get a readable URL? Example: http://www.python.org/dev/peps/qualname/ instead of http://www.python.org/dev/peps/pep-0395/ Other examples: 305: csv 450: statistics 3156: asyncio An identifier must only contain lower case letters (a-z), digits (0-9), dot (.) and underscore (_). It should be short (max. 20 characters)... and unique. Regex: ^[a-z0-9._]{1,20}$. For draft PEP, the identifier may change. If it's an issue (dead URL), we may allow multiple identifiers (ex: add also "tulip" for 3156), or only add an identifier to accepted PEPs. I don't know yet how it can be implemented at server side. Victor

12 18

Re: [Python-Dev] [Python-checkins] cpython: Issue #19209: fix structseq test
by Nick Coghlan 12 Oct '13

12 Oct '13

On 12 Oct 2013 09:42, "christian.heimes" <python-checkins(a)python.org> wrote: > > http://hg.python.org/cpython/rev/89e405e6a7a9 > changeset: 86218:89e405e6a7a9 > parent: 86216:29c4a6a11e76 > user: Christian Heimes <christian(a)cheimes.de> > date: Sat Oct 12 01:38:52 2013 +0200 > summary: > Issue #19209: fix structseq test > > files: > Lib/test/test_structseq.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/test_structseq.py b/Lib/test/test_structseq.py > --- a/Lib/test/test_structseq.py > +++ b/Lib/test/test_structseq.py > @@ -38,7 +38,7 @@ > # os.stat() gives a complicated struct sequence. > st = os.stat(__file__) > rep = repr(st) > - self.assertTrue(rep.startswith(os.name + ".stat_result")) > + self.assertTrue(rep.startswith("os.stat_result")) If stat results can be pickled, this suggests a possible issue with unpickling pickles generated with older versions of Python. Cheers, Nick. > self.assertIn("st_mode=", rep) > self.assertIn("st_ino=", rep) > self.assertIn("st_dev=", rep) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

2 1

Summary of Python tracker Issues
by Python tracker 12 Oct '13

12 Oct '13

ACTIVITY SUMMARY (2013-10-04 - 2013-10-11) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4240 (-16) closed 26757 (+75) total 30997 (+59) Open issues with patches: 1960 Issues opened (37) ================== #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 reopened by joru #18716: Deprecate the formatter module http://bugs.python.org/issue18716 reopened by brett.cannon #19167: sqlite3 cursor.description varies across Linux (3.3.1), Win32 http://bugs.python.org/issue19167 opened by mpb #19170: telnetlib: use selectors http://bugs.python.org/issue19170 opened by neologix #19172: selectors: add keys() method http://bugs.python.org/issue19172 opened by neologix #19173: Expose Queue maxsize parameter to multiprocessing.Pool class http://bugs.python.org/issue19173 opened by noxdafox #19175: Erroneous reference to "integer" in format string grammar http://bugs.python.org/issue19175 opened by davidchambers #19176: DeprecationWarning for doctype() method when subclassing _elem http://bugs.python.org/issue19176 opened by vadmium #19179: doc bug: confusing table of values http://bugs.python.org/issue19179 opened by georg.brandl #19180: some RFC references could be updated http://bugs.python.org/issue19180 opened by georg.brandl #19183: PEP 456 Secure and interchangeable hash algorithm http://bugs.python.org/issue19183 opened by christian.heimes #19184: dis module has incorrect docs for RAISE_VARARGS http://bugs.python.org/issue19184 opened by nedbat #19185: Allow multiprocessing Pool initializer to return values http://bugs.python.org/issue19185 opened by noxdafox #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 opened by athompson #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 opened by serhiy.storchaka #19191: os.path.splitext in windows , a little question http://bugs.python.org/issue19191 opened by xiaowei.py #19192: Move test_current_time from test_xmlrpc_net to test_xmlrpc http://bugs.python.org/issue19192 opened by vajrasky #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 opened by serhiy.storchaka #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 opened by serhiy.storchaka #19201: lzma and 'x' mode open http://bugs.python.org/issue19201 opened by oylenshpeegul #19202: Additions to docs http://bugs.python.org/issue19202 opened by Esa.Peuha #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 opened by serhiy.storchaka #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 opened by serhiy.storchaka #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 opened by christian.heimes #19206: Support disabling file I/O when doing traceback formatting http://bugs.python.org/issue19206 opened by itamarst #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 opened by serhiy.storchaka #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 opened by christian.heimes #19213: platform.linux_distribution detects Oracle Linux as Red Hat En http://bugs.python.org/issue19213 opened by javawizard #19214: shutil.make_archive should recognize extensions in filenames http://bugs.python.org/issue19214 opened by andreas-h #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 opened by christian.heimes #19217: Calling assertEquals for moderately long list takes too long http://bugs.python.org/issue19217 opened by Jacek.Bzdak #19218: Use of MutableMapping in os module slows down interpreter star http://bugs.python.org/issue19218 opened by eric.snow #19219: speed up marshal.loads() http://bugs.python.org/issue19219 opened by pitrou #19222: gzip and 'x' mode open http://bugs.python.org/issue19222 opened by oylenshpeegul #19223: bz2 and 'x' mode open http://bugs.python.org/issue19223 opened by oylenshpeegul #19224: Make hash(None) consistent among processes http://bugs.python.org/issue19224 opened by hongqn #19225: lack of PyExc_BufferError doc http://bugs.python.org/issue19225 opened by xwhhsprings Most recent 15 issues with no replies (15) ========================================== #19223: bz2 and 'x' mode open http://bugs.python.org/issue19223 #19222: gzip and 'x' mode open http://bugs.python.org/issue19222 #19214: shutil.make_archive should recognize extensions in filenames http://bugs.python.org/issue19214 #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 #19184: dis module has incorrect docs for RAISE_VARARGS http://bugs.python.org/issue19184 #19180: some RFC references could be updated http://bugs.python.org/issue19180 #19179: doc bug: confusing table of values http://bugs.python.org/issue19179 #19176: DeprecationWarning for doctype() method when subclassing _elem http://bugs.python.org/issue19176 #19157: ipaddress.IPv6Network.hosts function omits network and broadca http://bugs.python.org/issue19157 Most recent 15 issues waiting for review (15) ============================================= #19224: Make hash(None) consistent among processes http://bugs.python.org/issue19224 #19219: speed up marshal.loads() http://bugs.python.org/issue19219 #19218: Use of MutableMapping in os module slows down interpreter star http://bugs.python.org/issue19218 #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 #19207: Improve cross-references in os documentation http://bugs.python.org/issue19207 #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 #19204: Improve cross-references in urllib documentation http://bugs.python.org/issue19204 #19203: Improve cross-references in curses howto http://bugs.python.org/issue19203 #19202: Additions to docs http://bugs.python.org/issue19202 #19201: lzma and 'x' mode open http://bugs.python.org/issue19201 #19198: Improve cross-references in cgi documentation http://bugs.python.org/issue19198 #19197: Improve cross-references in shlex documentation http://bugs.python.org/issue19197 #19192: Move test_current_time from test_xmlrpc_net to test_xmlrpc http://bugs.python.org/issue19192 #19189: Improve cross-references in pickle documentation. http://bugs.python.org/issue19189 Top 10 most discussed issues (10) ================================= #19219: speed up marshal.loads() http://bugs.python.org/issue19219 31 msgs #3982: support .format for bytes http://bugs.python.org/issue3982 20 msgs #12892: UTF-16 and UTF-32 codecs should reject (lone) surrogates http://bugs.python.org/issue12892 17 msgs #19209: Remove import copyreg from os module http://bugs.python.org/issue19209 14 msgs #19172: selectors: add keys() method http://bugs.python.org/issue19172 12 msgs #19183: PEP 456 Secure and interchangeable hash algorithm http://bugs.python.org/issue19183 11 msgs #19154: AttributeError: 'NoneType' in http/client.py when using select http://bugs.python.org/issue19154 10 msgs #19216: stat cache for import bootstrap http://bugs.python.org/issue19216 10 msgs #19142: Cross-compile fails trying to execute foreign pgen on build ho http://bugs.python.org/issue19142 9 msgs #18758: Fix internal references in the documentation http://bugs.python.org/issue18758 7 msgs Issues closed (71) ================== #1215: documentation doesn't say that you can't handle C segfaults fr http://bugs.python.org/issue1215 closed by python-dev #11585: Documentation 1.8 shows Python 2 example http://bugs.python.org/issue11585 closed by georg.brandl #11789: Extend upon metaclass/type class documentation, here: zope.int http://bugs.python.org/issue11789 closed by georg.brandl #11807: Documentation of add_subparsers lacks information about parame http://bugs.python.org/issue11807 closed by python-dev #12162: Documentation about re \number http://bugs.python.org/issue12162 closed by georg.brandl #12350: Improve stat_result.st_blocks and st_blksize documentation http://bugs.python.org/issue12350 closed by python-dev #13867: misleading comment in weakrefobject.h http://bugs.python.org/issue13867 closed by python-dev #13951: Document that Seg Fault in .so called by ctypes causes the int http://bugs.python.org/issue13951 closed by python-dev #14101: example function in tertools.count docstring is misindented http://bugs.python.org/issue14101 closed by georg.brandl #14224: packaging: path description of resources is mixed up http://bugs.python.org/issue14224 closed by georg.brandl #15172: Document nasm-2.10.01 as required version for openssl http://bugs.python.org/issue15172 closed by georg.brandl #15213: _PyOS_URandom documentation http://bugs.python.org/issue15213 closed by python-dev #15228: os.utime() docs not clear on behavior on nonexistant files http://bugs.python.org/issue15228 closed by python-dev #15264: PyErr_SetFromErrnoWithFilenameObject() undocumented http://bugs.python.org/issue15264 closed by georg.brandl #15329: clarify which deque methods are thread-safe http://bugs.python.org/issue15329 closed by rhettinger #15432: gzip.py: mtime argument only since python 2.7 http://bugs.python.org/issue15432 closed by python-dev #15455: index entries not showing up in glossary http://bugs.python.org/issue15455 closed by georg.brandl #15805: Add stdout redirection tool to contextlib http://bugs.python.org/issue15805 closed by rhettinger #15863: Fine-grained info about Python versions which support changes http://bugs.python.org/issue15863 closed by georg.brandl #15964: SyntaxError in asdl when building 2.7 with system Python 3 http://bugs.python.org/issue15964 closed by eli.bendersky #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which is http://bugs.python.org/issue16742 closed by python-dev #17725: English mistake in Extending and Embedding Python doc page. http://bugs.python.org/issue17725 closed by python-dev #17745: "packaging" no longer planned to be included http://bugs.python.org/issue17745 closed by georg.brandl #18529: Use long dash http://bugs.python.org/issue18529 closed by georg.brandl #18577: lru_cache enhancement: lru_timestamp helper function http://bugs.python.org/issue18577 closed by rhettinger #18646: Improve tutorial entry on 'Lambda Forms'. http://bugs.python.org/issue18646 closed by python-dev #18664: occasional test_threading failure http://bugs.python.org/issue18664 closed by haypo #18764: The pdb print command prints repr instead of str in python3 http://bugs.python.org/issue18764 closed by r.david.murray #18785: Add get_body and iter_attachments to provisional email API http://bugs.python.org/issue18785 closed by r.david.murray #18790: incorrect text in argparse add_help example http://bugs.python.org/issue18790 closed by berker.peksag #18860: Add content manager API to email package http://bugs.python.org/issue18860 closed by r.david.murray #18890: Add a raw_data_manager content manager to the email package. http://bugs.python.org/issue18890 closed by r.david.murray #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 closed by python-dev #18948: deliberately crashing tests should prevent core dumps http://bugs.python.org/issue18948 closed by pitrou #18972: Use argparse in email example scripts http://bugs.python.org/issue18972 closed by serhiy.storchaka #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 closed by rhettinger #19067: Built-in range docstrings are not PEP-8 compatible http://bugs.python.org/issue19067 closed by ezio.melotti #19068: Some built-in complex docstrings are not PEP-8 compatible http://bugs.python.org/issue19068 closed by ezio.melotti #19074: Add PySide to GUI FAQ http://bugs.python.org/issue19074 closed by ezio.melotti #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 closed by pitrou #19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots http://bugs.python.org/issue19106 closed by eli.bendersky #19129: 6.2.1. Regular Expression Syntax flags http://bugs.python.org/issue19129 closed by georg.brandl #19148: Minor issues with Enum docs http://bugs.python.org/issue19148 closed by ezio.melotti #19156: Enum helper functions test-coverage http://bugs.python.org/issue19156 closed by python-dev #19158: BoundedSemaphore.release() subject to races http://bugs.python.org/issue19158 closed by tim.peters #19161: collections Counter handles nan strangely http://bugs.python.org/issue19161 closed by rhettinger #19166: Unusued variable in test_keys in Lib/test/test_dict.py http://bugs.python.org/issue19166 closed by ezio.melotti #19168: pprint.pprint(..., compact=True) not implemented http://bugs.python.org/issue19168 closed by ezio.melotti #19169: random.py : simple tidying http://bugs.python.org/issue19169 closed by rhettinger #19171: pow() improvement on longs http://bugs.python.org/issue19171 closed by tim.peters #19174: Add range to future_builtins http://bugs.python.org/issue19174 closed by rhettinger #19177: Link to "TLS (Transport Layer Security) and SSL (Secure Socket http://bugs.python.org/issue19177 closed by python-dev #19178: Entries for "module" and "package" in glossary http://bugs.python.org/issue19178 closed by python-dev #19181: ftp.cwi.nl used as example does not exist anymore http://bugs.python.org/issue19181 closed by python-dev #19182: Socket leak in logging.handlers http://bugs.python.org/issue19182 closed by vinay.sajip #19187: Use a set for interned strings http://bugs.python.org/issue19187 closed by rhettinger #19188: Add PySet_CheckExact() http://bugs.python.org/issue19188 closed by rhettinger #19190: Improve cross-references in builtins documentation. http://bugs.python.org/issue19190 closed by serhiy.storchaka #19193: Improve cross-references in tutorial http://bugs.python.org/issue19193 closed by serhiy.storchaka #19194: Improve cross-references in fcntl documentation http://bugs.python.org/issue19194 closed by serhiy.storchaka #19195: Improve cross-references in C API http://bugs.python.org/issue19195 closed by serhiy.storchaka #19196: Improve cross-references in distutils documentation http://bugs.python.org/issue19196 closed by serhiy.storchaka #19199: Remove PyThreadState.tick_counter field http://bugs.python.org/issue19199 closed by python-dev #19200: grammar in forkserver docs http://bugs.python.org/issue19200 closed by python-dev #19208: Bas64.decodestring() returns data instead of throwing exceptio http://bugs.python.org/issue19208 closed by r.david.murray #19210: Unicode Objects in Tuples http://bugs.python.org/issue19210 closed by pitrou #19211: from relative_module import seems to import wrong module http://bugs.python.org/issue19211 closed by r.david.murray #19212: Spam http://bugs.python.org/issue19212 closed by r.david.murray #19215: StringIO.StringIO('foo').readline(0) == 'foo' http://bugs.python.org/issue19215 closed by r.david.murray #19220: Outdated unicodedata.__doc__ http://bugs.python.org/issue19220 closed by python-dev #19221: Upgrade to Unicode 6.3.0 http://bugs.python.org/issue19221 closed by python-dev

2 1

Python IDEL issue
by Hu, Hao (NSN - CN/Beijing) 12 Oct '13

12 Oct '13

Hi, When I open IDLE(Python GUI) from the windows start menu, it always open Python Shell. When I right click on a *.py file, I cannot see the menu item "open with IDLE". Even after I uninstall and reinstall Python several times with different versions, the issue remains. I guess it is because something prerequisite is not installed on my computer. But do you know what is it?

2 1

Re: [Python-Dev] [Python-checkins] peps: Revise PEP 453 to be docs-only for 2.7 & 3.3
by Terry Reedy 11 Oct '13

11 Oct '13

On 10/10/2013 10:48 AM, nick.coghlan wrote: > http://hg.python.org/peps/rev/405b80d54b7d > changeset: 5188:405b80d54b7d > user: Nick Coghlan <ncoghlan(a)gmail.com> > date: Fri Oct 11 00:47:47 2013 +1000 > summary: > Revise PEP 453 to be docs-only for 2.7 & 3.3 > > - all functional changes are now 3.4 only > - still proposes docs changes for 2.7 & 3.3 > - notes current effort to create a Windows installer for pip > - notes possibility of a future PEP to provide a combined > CPython 2.7, pip and Python Launcher installer from python.org > +`Installing Python Modules <http://docs.python.org/3/install>`__ guide in > +Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of ``pip`` > +as the default installer for Python packages, and that appropriate technical > +changes be made in Python 3.4 to provide ``pip`` by default in support of > +that recommendation. Does one need a separate installation of pip for each Python version or is pip 'separate' enough that one installation works for all Python versions? Is the answer to this the same on all systems? If the answer is 'one' and 'yes', the 2.7 and 3.3 docs should say that installing 3.4 also installs pip and that this is all one needs to do. > +While this PEP only proposes documentation changes for Python 2.7, once > +``pip`` has a Windows installer available, a separate PEP will be created > +and submitted proposing the creation and distribution of aggregate installers > +for future CPython 2.7 maintenance releases that combine the CPython, > +``pip`` and Python Launcher for Windows installers into a single download > +(the separate downloads would still remain available - the aggregate > +installers would be provided as a convenience, and as a clear indication > +of the recommended operating environment for Python in Windows systems). If the combined installer is an optional convenience, I would not think a PEP necessary. I am assuming that the combined installer would not add a module to /lib any more than two separate installers would, and hence would not be adding a feature to 2.7 itself. -- Terry Jan Reedy

2 1

Fix IO Encoding issue for embedded Python under Windows - Issue16129
by Bastien Montagne 11 Oct '13

11 Oct '13

Hi Py Devs, As this is my first mail here, first a short presentation: my name is Bastien Montagne (aka mont29), I live in France, and I’ve been using Python for a fair amount of years now in various projects, amongst which Blender 3D software. I’m also a Blender dev (C & Py) since a few years. Now to the topic, to summarize http://bugs.python.org/issue16129 , an application dynamically linking against Python under MS-Windows has no way to control Python's environment, and hence cannot set PYTHONIOENCODING to get another encoding than default Windows' console (which is usually 8bit, like cp437). This is really annoying under Blender, as we use utf-8 strings, users having e.g. "special" chars in their file paths keep getting python errors instead of nice prints… Following Nick Coghlan’s suggestions, I made a patch which solves that issue by adding Py_SetStandardStreamEncoding() to the API, which allows embedding code to directly specify standard IO encodings. So now I’m waiting for reviews & comments from py devs, in the hope that this can make it into python code before 3.4 feature freeze (patch: http://bugs.python.org/file31985/setstdio.diff ). Best regards, Bastien

2 1

Reduce memory footprint of Python
by Victor Stinner 11 Oct '13

11 Oct '13

Hi, Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. First, I noticed that linecache can allocate more than 2 MB. What do you think of adding a registry of "clear cache" functions? For exemple, re.purge() and linecache.clearcache(). gc.collect() clears free lists. I don't know if gc.collect() should be related to this new registy (clear all caches) or not. The dictionary of interned Unicode strings can be large: up to 1.5 MB (with +30,000 strings). Just the dictionary, excluding size of strings. Is the size normal or not? Using tracemalloc, this dictionary is usually to largest memory block. unittest doesn't look to release memory (the TestCase class) after the execution of a test. test_import.test_module_with_large_stack() creates a large Python module and import it, but it does not unload it. Should I open a separated issue for each idea to track them in the bug tracker, or a global issue? Victor

17 42

PEP 457: Syntax For Positional-Only Parameters
by Larry Hastings 10 Oct '13

10 Oct '13

I've contributed a new PEP to humanity. I include the RST for your reading pleasure below, but you can also read it online here: http://www.python.org/dev/peps/pep-0457/ Discuss, //arry/ ----- PEP: 457 Title: Syntax For Positional-Only Parameters Version: $Revision$ Last-Modified: $Date$ Author: Larry Hastings <larry(a)hastings.org> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 08-Oct-2013 ======== Overview ======== This PEP proposes a syntax for positional-only parameters in Python. Positional-only parameters are parameters without an externally-usable name; when a function accepting positional-only parameters is called, positional arguments are mapped to these parameters based solely on their position. ========= Rationale ========= Python has always supported positional-only parameters. Early versions of Python lacked the concept of specifying parameters by name, so naturally all parameters were positional-only. This changed around Python 1.0, when all parameters suddenly became positional-or-keyword. But, even in current versions of Python, many CPython "builtin" functions still only accept positional-only arguments. Functions implemented in modern Python can accept an arbitrary number of positional-only arguments, via the variadic ``*args`` parameter. However, there is no Python syntax to specify accepting a specific number of positional-only parameters. Put another way, there are many builtin functions whose signatures are simply not expressable with Python syntax. This PEP proposes a backwards-compatible syntax that should permit implementing any builtin in pure Python code. ----------------------------------------------------- Positional-Only Parameter Semantics In Current Python ----------------------------------------------------- There are many, many examples of builtins that only accept positional-only parameters. The resulting semantics are easily experienced by the Python programmer--just try calling one, specifying its arguments by name:: >>> pow(x=5, y=3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: pow() takes no keyword arguments In addition, there are some functions with particularly interesting semantics: * ``range()``, which accepts an optional parameter to the *left* of its required parameter. [#RANGE]_ * ``dict()``, whose mapping/iterator parameter is optional and semantically must be positional-only. Any externally visible name for this parameter would occlude that name going into the ``**kwarg`` keyword variadic parameter dict! [#DICT]_ Obviously one can simulate any of these in pure Python code by accepting ``(*args, **kwargs)`` and parsing the arguments by hand. But this results in a disconnect between the Python function's signature and what it actually accepts, not to mention the work of implementing said argument parsing. ========== Motivation ========== This PEP does not propose we implement positional-only parameters in Python. The goal of this PEP is simply to define the syntax, so that: * Documentation can clearly, unambiguously, and consistently express exactly how the arguments for a function will be interpreted. * The syntax is reserved for future use, in case the community decides someday to add positional-only parameters to the language. * Argument Clinic can use a variant of the syntax as part of its input when defining the arguments for built-in functions. ================================================================= The Current State Of Documentation For Positional-Only Parameters ================================================================= The documentation for positional-only parameters is incomplete and inconsistent: * Some functions denote optional groups of positional-only arguments by enclosing them in nested square brackets. [#BORDER]_ * Some functions denote optional groups of positional-only arguments by presenting multiple prototypes with varying numbers of arguments. [#SENDFILE]_ * Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_ One more important idea to consider: currently in the documentation there's no way to tell whether a function takes positional-only parameters. ``open()`` accepts keyword arguments, ``ord()`` does not, but there is no way of telling just by reading the documentation that this is true. ==================== Syntax And Semantics ==================== From the "ten-thousand foot view", and ignoring ``*args`` and ``**kwargs`` for now, the grammar for a function definition currently looks like this:: def name(positional_or_keyword_parameters, *, keyword_only_parameters): Building on that perspective, the new syntax for functions would look like this:: def name(positional_only_parameters, /, positional_or_keyword_parameters, *, keyword_only_parameters): All parameters before the ``/`` are positional-only. If ``/`` is not specified in a function signature, that function does not accept any positional-only parameters. Positional-only parameters can be optional, but the mechanism is significantly different from positional-or-keyword or keyword-only parameters. Positional-only parameters don't accept default values. Instead, positional-only parameters can be specified in optional "groups". Groups of parameters are surrounded by square brackets, like so:: def addch([y, x,] ch, [attr], /): Positional-only parameters that are not in an option group are "required" positional-only parameters. All "required" positional-only parameters must be contiguous. Parameters in an optional group accept arguments in a group; you must provide arguments either for all of the them or for none of them. Using the example of ``addch()`` above, you could not call ``addch()`` in such a way that ``x`` was specified but ``y`` was not (and vice versa). The mapping of positional parameters to optional groups is done based on fitting the number of parameters to groups. Based on the above definition, ``addch()`` would assign arguments to parameters in the following way: +-------------------+------------------------------+ |Number of arguments|Parameter assignment | +-------------------+------------------------------+ |0 |*raises an exception* | +-------------------+------------------------------+ |1 |``ch`` | +-------------------+------------------------------+ |2 |``ch``, ``attr`` | +-------------------+------------------------------+ |3 |``y``, ``x``, ``ch`` | +-------------------+------------------------------+ |4 |``y``, ``x``, ``ch``, ``attr``| +-------------------+------------------------------+ |5 or more |*raises an exception* | +-------------------+------------------------------+ More semantics of positional-only parameters: * Although positional-only parameter technically have names, these names are internal-only; positional-only parameters are *never* externally addressable by name. (Similarly to ``*args`` and ``**kwargs``.) * It's possible to nest option groups. * If there are no required parameters, all option groups behave as if they're to the right of the required parameter group. * For clarity and consistency, the comma for a parameter always comes immediately after the parameter name. It's a syntax error to specify a square bracket between the name of a parameter and the following comma. (This is far more readable than putting the comma outside the square bracket, particularly for nested groups.) * If there are arguments after the ``/``, then you must specify a comma after the ``/``, just as there is a comma after the ``*`` denoting the shift to keyword-only parameters. * This syntax has no effect on ``*args`` or ``**kwargs``. It's possible to specify a function prototype where the mapping of arguments to parameters is ambiguous. Consider:: def range([start,] stop, [range], /): Python disambiguates these situations by preferring optional groups to the *left* of the required group. ====================== Additional Limitations ====================== Argument Clinic uses a form of this syntax for specifying builtins. It imposes further limitations that are theoretically unnecessary but make the implementation easier. Specifically: * A function that has positional-only parameters currently cannot have any other kind of parameter. (This will probably be relaxed slightly in the near future.) * Multiple option groups on either side of the required positional-only parameters must be nested, with the nesting getting deeper the further away the group is from the required positional-parameter group. Put another way: all the left-brackets for option groups to the left of the required group must be specified contiguously, and all the right-brackets for option groups to the right of the required group must be specified contiguously. ============================== Notes For A Future Implementor ============================== If we decide to implement positional-only parameters in a future version of Python, we'd have to do some additional work to preserve their semantics. The problem: how do we inform a parameter that no value was passed in for it when the function was called? The obvious solution: add a new singleton constant to Python that is passed in when a parameter is not mapped to an argument. I propose that the value be called called ``undefined``, and be a singleton of a special class called ``Undefined``. If a positional-only parameter did not receive an argument when called, its value would be set to ``undefined``. But this raises a further problem. How do can we tell the difference between "this positional-only parameter did not receive an argument" and "the caller passed in ``undefined`` for this parameter"? It'd be nice to make it illegal to pass ``undefined`` in as an argument to a function--to, say, raise an exception. But that would slow Python down, and the "consenting adults" rule appears applicable here. So making it illegal should probably be strongly discouraged but not outright prevented. However, it should be allowed (and encouraged) for user functions to specify ``undefined`` as a default value for parameters. ==================== Unresolved Questions ==================== There are three types of parameters in Python: 1. positional-only parameters, 2. positional-or-keyword parameters, and 3. keyword-only parameters. Python allows functions to have both 2 and 3. And some builtins (e.g. range) have both 1 and 3. Does it make sense to have functions that have both 1 and 2? Or all of the above? ====== Thanks ====== Credit for the use of '/' as the separator between positional-only and positional-or-keyword parameters goes to Guido van Rossum, in a proposal from 2012. [#GUIDO]_ Credit for making left option groups higher precedence goes to Nick Coghlan. (Conversation in person at PyCon US 2013.) .. [#DICT] http://docs.python.org/3/library/stdtypes.html#dict .. [#RANGE] http://docs.python.org/3/library/functions.html#func-range .. [#BORDER] http://docs.python.org/3/library/curses.html#curses.window.border .. [#SENDFILE] http://docs.python.org/3/library/os.html#os.sendfile .. [#ADDCH] http://docs.python.org/3/library/curses.html#curses.window.addch .. [#GUIDO] Guido van Rossum, posting to python-ideas, March 2012: http://mail.python.org/pipermail/python-ideas/2012-March/014364.html and http://mail.python.org/pipermail/python-ideas/2012-March/014378.html and http://mail.python.org/pipermail/python-ideas/2012-March/014417.html ========= Copyright ========= This document has been placed in the public domain.

19 68

PEP 454: Add a new tracemalloc module (final version)
by Victor Stinner 10 Oct '13

10 Oct '13

Hi, I worked on the implementation of the tracemalloc module and its PEP 454. I consider that this third version of the PEP is ready for a final review. What should be done to finish the PEP? HTML version of the PEP 454: http://www.python.org/dev/peps/pep-0454/ Full documentation of the tracemalloc module, with examples, a short tutorial and the documentation of the command line (python -m tracemalloc): http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html I addressed all points of my own TODO list, even minor tasks. I still have some minor tasks, but only on the implementation, not on the API of the module. The API is designed to process easily data collected by tracemalloc and to be extensible. It should now be easy to write a GUI to display these data for example. API changes between versions 2 and 3 of the PEP: * Add Metric class: in version 2, a snapshot had an "user_data" attribute which was not defined, not even its type. Metrics are now formalized, displayed and can be compared. They are used to track the process memory, number of Python objects, size of memory traced by tracemalloc, etc. The API also allows you to add your own metrics. * Rewrite the "scheduler" API. I removed the restriction of a single task at the same time. tracemalloc now supports multiple tasks which have a new trigger: threshold on the traced memory. I added a new Task class. DisplayTopTask and TakeSnapshotTask classes inherit from Task, to inherit its methods and attributes. It is now possible to only schedule a task "repeat" times. * Add DisplayTop.display() method, more convinient than having to create a DisplayTopTask instance. * Remove Frame, Trace and TraceStats classes for efficiency. tracemalloc handles millions of such objects. Creating and serialization millions of such small objects could take up to 30 seconds. I was not really convinced that a class is required to store only two fields, a tuple is just fine for a debug module. * (and many other minor changes) I optimized the implementation which is now faster and releases the memory earlier thanks to a new memory pool (only used by the module). PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations read the C filename and line number. Using such tool to analyze Python memory allocations does not help because most memory block are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These tools analyze objects type and/or content. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. A hook can inspect Python internals to retrieve the Python tracebacks. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Compute the differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/dev/library/faulthandler.html>`_. The tracemalloc module has been written for CPython. Other implementations of Python may not provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can also be called to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores one frame. Use the ``set_traceback_limit()`` function to store more frames. Python memory blocks allocated in the ``tracemalloc`` module are also traced by default. Use ``add_exclude_filter(tracemalloc.__file__)`` to ignore these these memory allocations. At fork, the module is automatically disabled in the child process. Main Functions -------------- ``cancel_tasks()`` function: Cancel all scheduled tasks. See also the ``get_tasks()`` function. ``clear_traces()`` function: Clear all traces and statistics on Python memory allocations, and reset the ``get_arena_size()`` and ``get_traced_memory()`` counters. ``disable()`` function: Stop tracing Python memory allocations and cancel scheduled tasks. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. At fork, the module is automatically disabled in the child process. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_tasks()`` function: Get the list of scheduled tasks, list of ``Task`` instances. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. Trace Functions --------------- ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace of a memory block. Use the ``set_traceback_limit()`` function to change the limit. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. See also ``get_object_trace()`` and ``gc.get_referrers()`` functions. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. The function only returns the trace of the main memory block of the object. The *size* of the trace is smaller than the total size of the object if the object is composed by more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``get_trace()``, ``get_traces()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block as a ``(size: int, traceback)`` tuple where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_trace()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traces()`` function: Get all traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``(size: int, traceback)`` and *traceback* is a list of ``(filename: str, lineno: int)``. *traceback* can be empty, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_trace()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace of a memory block. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filter match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``clear_traces()`` function to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` method creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. See also the ``clear_filters()`` function. Metric Functions ---------------- The following functions can be used to add metrics to a snapshot, see the ``Snapshot.add_metric()`` method. ``get_allocated_blocks()`` function: Get the current number of allocated memory blocks. ``get_arena_size()`` function: Get the size in bytes of traced arenas. See also the ``get_pymalloc_stats()`` function. ``get_process_memory()`` function: Get the memory usage of the current process as a ``(rss: int, vms: int)`` tuple, *rss* is the "Resident Set Size" in bytes and *vms* is the size of the virtual memory in bytes Return ``None`` if the platform is not supported. ``get_pymalloc_stats()`` function: Get statistics on the ``pymalloc`` allocator as a dictionary. +---------------------+-------------------------------------------------------+ | Key | Description | +=====================+=======================================================+ | ``alignment`` | Alignment of addresses returned to the user. | +---------------------+-------------------------------------------------------+ | ``threshold`` | Small block threshold in bytes: pymalloc uses | | | PyMem_RawMalloc() for allocation greater than | | | threshold. | +---------------------+-------------------------------------------------------+ | ``nalloc`` | Number of times object malloc called | +---------------------+-------------------------------------------------------+ | ``arena_size`` | Arena size in bytes | +---------------------+-------------------------------------------------------+ | ``total_arenas`` | Number of calls to new_arena(): total number of | | | allocated arenas, including released arenas | +---------------------+-------------------------------------------------------+ | ``max_arenas`` | Maximum number of arenas | +---------------------+-------------------------------------------------------+ | ``arenas`` | Number of arenas currently allocated | +---------------------+-------------------------------------------------------+ | ``allocated_bytes`` | Number of bytes in allocated blocks | +---------------------+-------------------------------------------------------+ | ``available_bytes`` | Number of bytes in available blocks in used pools | +---------------------+-------------------------------------------------------+ | ``pool_size`` | Pool size in bytes | +---------------------+-------------------------------------------------------+ | ``free_pools`` | Number of unused pools | +---------------------+-------------------------------------------------------+ | ``pool_headers`` | Number of bytes wasted in pool headers | +---------------------+-------------------------------------------------------+ | ``quantization`` | Number of bytes in used and full pools wasted due to | | | quantization, i.e. the necessarily leftover space at | | | the ends of used and full pools. | +---------------------+-------------------------------------------------------+ | ``arena_alignment`` | Number of bytes for arena alignment padding | +---------------------+-------------------------------------------------------+ The function is not available if Python is compiled without ``pymalloc``. See also ``get_arena_size()`` and ``sys._debugmallocstats()`` functions. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module as a tuple: ``(size: int, free: int)``. * *size*: total size of bytes allocated by the module, including *free* bytes * *free*: number of free bytes available to store data ``get_unicode_interned()`` function: Get the size in bytes and the length of the dictionary of Unicode interned strings as a ``(size: int, length: int)`` tuple. The size is the size of the dictionary, excluding the size of strings. DisplayTop ---------- ``DisplayTop()`` class: Display the top of allocated memory blocks. ``display(count=10, group_by="line", cumulative=False, file=None, callback=None)`` method: Take a snapshot and display the top *count* biggest allocated memory blocks grouped by *group_by*. *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. Return the snapshot, a ``Snapshot`` instance. ``display_snapshot(snapshot, count=10, group_by="line", cumulative=False, file=None)`` method: Display a snapshot of memory blocks allocated by Python, *snapshot* is a ``Snapshot`` instance. ``display_top_diff(top_diff, count=10, file=None)`` method: Display differences between two ``GroupedStats`` instances, *top_diff* is a ``StatsDiff`` instance. ``display_top_stats(top_stats, count=10, file=None)`` method: Display the top of allocated memory blocks grouped by the ``GroupedStats.group_by`` attribute of *top_stats*, *top_stats* is a ``GroupedStats`` instance. ``average`` attribute: If ``True`` (default value), display the average size of memory blocks. ``color`` attribute: If ``True``, always use colors. If ``False``, never use colors. The default value is ``None``: use colors if the *file* parameter is a TTY device. ``compare_to_previous`` attribute: If ``True`` (default value), compare to the previous snapshot. If ``False``, compare to the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``'...'``. ``metrics`` attribute: If ``True`` (default value), display metrics: see ``Snapshot.metrics``. ``previous_top_stats`` attribute: Previous ``GroupedStats`` instance, or first ``GroupedStats`` instance if ``compare_to_previous`` is ``False``, used to display the differences between two snapshots. ``size`` attribute: If ``True`` (default value), display the size of memory blocks. DisplayTopTask -------------- ``DisplayTopTask(count=10, group_by="line", cumulative=False, file=sys.stdout, callback=None)`` class: Task taking temporary snapshots and displaying the top *count* memory allocations grouped by *group_by*. ``DisplayTopTask`` is based on the ``Task`` class and so inherit all attributes and methods, especially: * ``Task.cancel()`` * ``Task.schedule()`` * ``Task.set_delay()`` * ``Task.set_memory_threshold()`` Modify the ``display_top`` attribute to customize the display. ``display()`` method: Take a snapshot and display the top ``count`` biggest allocated memory blocks grouped by ``group_by`` using the ``display_top`` attribute. Return the snapshot, a ``Snapshot`` instance. ``callback`` attribute: *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. ``count`` attribute: Maximum number of displayed memory blocks. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of each trace, not only the most recent frame. The default value is ``False``. The option is ignored if the traceback limit is less than ``2``, see the ``get_traceback_limit()`` function. ``display_top`` attribute: Instance of ``DisplayTop``. ``file`` attribute: The top is written into *file*. ``group_by`` attribute: Determine how memory allocations are grouped: see ``Snapshot.top_by()`` for the available values. Filter ------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filter matchs the filename and line number, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filter matchs the filename, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the filter matchs the line number, ``False`` otherwise. ``match_traceback(traceback)`` method: Return ``True`` if the filter matchs the *traceback*, ``False`` otherwise. *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename :attr`pattern` at line number ``lineno``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or less than ``1``, it matches any line number. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class: Top of allocated memory blocks grouped by *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. The ``StatsDiff.differences`` list is not sorted: call the ``StatsDiff.sort`` method to sort the list. ``None`` values are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``group_by`` attribute: Determine how memory allocations were grouped: see ``Snapshot.top_by()`` for the available values. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* is a ``(size: int, count: int)`` tuple. See the ``Snapshot.top_by()`` method. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Metric ------ ``Metric(name: str, value: int, format: str)`` class: Value of a metric when a snapshot is created. ``name`` attribute: Name of the metric. ``value`` attribute: Value of the metric. ``format`` attribute: Format of the metric: * ``'int'``: a number * ``'percent'``: percentage, ``1.0`` means ``100%`` * ``'size'``: a size in bytes Snapshot -------- ``Snapshot(timestamp: datetime.datetime, pid: int, traces: dict=None, stats: dict=None, metrics: dict=None)`` class: Snapshot of traces and statistics on memory blocks allocated by Python. Use ``TakeSnapshotTask`` to take regulary snapshots. ``add_gc_metrics()`` method: Add a metric on garbage collector: * ``gc.objects``: total number of Python objects See the ``gc`` module. ``add_metric(name: str, value: int, format: str)`` method: Helper to add a ``Metric`` instance to ``Snapshot.metrics``. Return the newly created ``Metric`` instance. Raise an exception if the name is already present in ``Snapshot.metrics``. ``add_process_memory_metrics()`` method: Add metrics on the process memory: * ``process_memory.rss``: Resident Set Size * ``process_memory.vms``: Virtual Memory Size These metrics are only available if the ``get_process_memory()`` function is available on the platform. ``add_pymalloc_metrics()`` method: Add metrics on the Python memory allocator (``pymalloc``): * ``pymalloc.blocks``: number of allocated memory blocks * ``pymalloc.size``: size of ``pymalloc`` arenas * ``pymalloc.max_size``: maximum size of ``pymalloc`` arenas * ``pymalloc.allocated``: number of allocated bytes * ``pymalloc.free``: number of free bytes * ``pymalloc.fragmentation``: fragmentation percentage of the arenas These metrics are only available if Python is compiled in debug mode, except ``pymalloc.blocks`` which is always available. ``add_tracemalloc_metrics()`` method: Add metrics on the ``tracemalloc`` module: * ``tracemalloc.traced.size``: size of memory blocks traced by the ``tracemalloc`` module * ``tracemalloc.traced.max_size``: maximum size of memory blocks traced by the ``tracemalloc`` module * ``tracemalloc.traces``: number of traces of Python memory blocks * ``tracemalloc.module.size``: total size of bytes allocated by the ``tracemalloc`` module, including free bytes * ``tracemalloc.module.free``: number of free bytes available for the ``tracemalloc`` module * ``tracemalloc.module.fragmentation``: percentage of fragmentation of the memory allocated by the ``tracemalloc`` module * ``tracemalloc.arena_size``: size of traced arenas ``tracemalloc.traces`` metric is only present if the snapshot was created with traces. ``add_unicode_metrics()`` method: Add metrics on the Unicode interned strings: * ``unicode_interned.size``: size of the dictionary, excluding size of strings * ``unicode_interned.len``: length of the dictionary ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False, metrics=True)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *traces* is ``True``, ``get_traces`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. If *metrics* is ``True``, fill ``Snapshot.metrics`` with metrics using the following methods: * ``add_gc_metrics`` * ``add_process_memory_metrics`` * ``add_pymalloc_metrics`` * ``add_tracemalloc_metrics`` * ``add_unicode_metrics`` If *metrics* is ``False``, ``Snapshot.metrics`` is set to an empty dictionary. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot. See the the ``enable()`` function. ``get_metric(name, default=None)`` method: Get the value of the metric called *name*. Return *default* if the metric does not exist. ``load(filename, traces=True)`` classmethod: Load a snapshot from a file. If *traces* is ``False``, don't load traces. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(str, int)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is less than ``2``. ``write(filename)`` method: Write the snapshot into a file. ``metrics`` attribute: Dictionary storing metrics read when the snapshot was created: ``{name (str): metric}`` where *metric* type is ``Metric``. ``pid`` attribute: Identifier of the process which created the snapshot, result of ``os.getpid()``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in a trace of a memory block allocated by Python. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. StatsDiff --------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. The ``GroupedStats.compare_to`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest difference to the smallest difference. Sort by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the ``Snapshot.top_by()`` method. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Task ---- ``Task(func, *args, **kw)`` class: Task calling ``func(*args, **kw)``. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes, or after *delay* seconds. ``call()`` method: Call ``func(*args, **kw)`` and return the result. ``cancel()`` method: Cancel the task. Do nothing if the task is not scheduled. ``get_delay()`` method: Get the delay in seconds. If the delay is ``None``, the timer is disabled. ``get_memory_threshold()`` method: Get the threshold of the traced memory. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes. The memory threshold is disabled if *threshold* is ``None``. See also the ``set_memory_threshold()`` method and the ``get_traced_memory()`` function. ``schedule(repeat: int=None)`` method: Schedule the task *repeat* times. If *repeat* is ``None``, the task is rescheduled after each call until it is cancelled. If the method is called twice, the task is rescheduled with the new *repeat* parameter. The task must have a memory threshold or a delay: see ``set_delay()`` and ``set_memory_threshold()`` methods. The ``tracemalloc`` must be enabled to schedule a task: see the ``enable`` function. The task is cancelled if the ``call()`` method raises an exception. The task can be cancelled using the ``cancel()`` method or the ``cancel_tasks()`` function. ``set_delay(seconds: int)`` method: Set the delay in seconds before the task will be called. Set the delay to ``None`` to disable the timer. The timer is based on the Python memory allocator, it is not real time. The task is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. The timer has a resolution of 1 second. The task is rescheduled if it was scheduled. ``set_memory_threshold(size: int)`` method: Set the threshold of the traced memory. When scheduled, the task is called when the traced memory is increased or decreased by more than *threshold* bytes. Set the threshold to ``None`` to disable it. The task is rescheduled if it was scheduled. See also the ``get_memory_threshold()`` method and the ``get_traced_memory()`` function. ``func`` attribute: Function, callable object. ``func_args`` attribute: Function arguments, ``tuple``. ``func_kwargs`` attribute: Function keyword arguments, ``dict``. It can be ``None``. TakeSnapshotTask ---------------- ``TakeSnapshotTask(filename_template: str="tracemalloc-$counter.pickle", traces: bool=False, metrics: bool=True, callback: callable=None)`` class: Task taking snapshots of Python memory allocations and writing them into files. ``TakeSnapshotTask`` is based on the ``Task`` class and so inherit all attributes and methods, especially: * ``Task.cancel()`` * ``Task.schedule()`` * ``Task.set_delay()`` * ``Task.set_memory_threshold()`` ``take_snapshot()`` method: Take a snapshot and write it into a file. Return ``(snapshot, filename)`` where *snapshot* is a ``Snapshot`` instance and filename type is ``str``. ``callback`` attribute: *callback* is an optional callable object which can be used to add metrics to a snapshot. It is called with only one parameter: the newly created snapshot instance. Use the ``Snapshot.add_metric()`` method to add new metric. ``filename_template`` attribute: Template to create a filename. The template supports the following variables: * ``$pid``: identifier of the current process * ``$timestamp``: current date and time * ``$counter``: counter starting at 1 and incremented at each snapshot, formatted as 4 decimal digits The default template is ``'tracemalloc-$counter.pickle'``. ``metrics`` attribute: Parameter passed to the ``Snapshot.create()`` function. ``traces`` attribute: Parameter passed to the ``Snapshot.create()`` function. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain.

1 2

Re: [Python-Dev] cpython (3.3): Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in at least
by Georg Brandl 10 Oct '13

10 Oct '13

Am 07.10.2013 20:41, schrieb antoine.pitrou: > http://hg.python.org/cpython/rev/2aee9c41bb79 > changeset: 86140:2aee9c41bb79 > branch: 3.3 > parent: 86136:b4444d16e333 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Mon Oct 07 20:38:51 2013 +0200 > summary: > Fix macro expansion of _PyErr_OCCURRED(), and make sure to use it in > at least one place so as to avoid regressions. If it was broken and never used anyway (and is an internal macro), why not remove it completely? Georg

2 1

Python-Dev October 2013