Mailman 3 September 2013 - Python-Dev

Pre-PEP: Redesigning extension modules
by Stefan Behnel Sept. 8, 2013

Sept. 8, 2013

Hi, this has been subject to a couple of threads on python-dev already, for example: http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986 http://thread.gmane.org/gmane.comp.python.devel/141037/focus=141046 It originally came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Here's an initial attempt at a PEP for it. It is based on the (unfinished) ModuleSpec PEP, which is being discussed on the import-sig mailing list. http://mail.python.org/pipermail/import-sig/2013-August/000688.html Stefan PEP: 4XX Title: Redesigning extension modules Version: $Revision$ Last-Modified: $Date$ Author: Stefan Behnel <stefan_ml at behnel.de> BDFL-Delegate: ??? Discussions-To: ??? Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2013 Python-Version: 3.4 Post-History: 23-Aug-2013 Resolution: Abstract ======== This PEP proposes a redesign of the way in which extension modules interact with the interpreter runtime. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve them by bringing extension modules closer to the way Python modules behave. An implication of this PEP is that extension modules can use arbitrary types for their module implementation and are no longer restricted to types.ModuleType. This makes it easy to support properties at the module level and to safely store arbitrary global state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Motivation ========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its fully qualified module name, FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it is not trivial to correctly add the module to sys.modules either. This is specifically a problem for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or reloading and it is neither easy nor efficient with the current infrastructure to support these features. This PEP also addresses these issues. The current process =================== Currently, extension modules export an initialisation function named "PyInit_modulename", named after the file name of the shared library. This function is executed by the import machinery and must return either NULL in the case of an exception, or a fully initialised module object. The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef struct. It then continues to initialise it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the FQMN of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The main problem in this process is the missing support for passing state into the module init function, and for safely passing state through to the module creation code. The proposal ============ The current extension module initialisation will be deprecated in favour of a new initialisation scheme. Since the current scheme will continue to be available, existing code will continue to work unchanged, including binary compatibility. Extension modules that support the new initialisation scheme must export a new public symbol "PyModuleCreate_modulename", where "modulename" is the name of the shared library. This mimics the previous naming convention for the "PyInit_modulename" function. This symbol must resolve to a C function with the following signature:: PyObject* (*PyModuleTypeCreateFunction)(PyObject* module_spec) The "module_spec" argument receives a "ModuleSpec" instance, as defined in PEP 4XX (FIXME). (All names are obviously up for debate and bike-shedding at this point.) When called, this function must create and return a type object, either a Python class or an extension type that is allocated on the heap. This type will be instantiated as module instance by the importer. There is no requirement for this type to be exactly or a subtype of types.ModuleType. Any type can be returned. This follows the current support for allowing arbitrary objects in sys.modules and makes it easier for extension modules to define a type that exactly matches their needs for holding module state. The constructor of this type must have the following signature:: def __init__(self, module_spec): The "module_spec" argument receives the same object as the one passed into the module type creation function. Implementation ============== XXX - not started Reloading and Sub-Interpreters ============================== To "reload" an extension module, the module create function is executed again and returns a new module type. This type is then instantiated as by the original module loader and replaces the previous entry in sys.modules. Once the last references to the previous module and its type are gone, both will be subject to normal garbage collection. Sub-interpreter support is an inherent property of the design. During import in the sub-interpreter, the module create function is executed and returns a new module type that is local to the sub-interpreter. Both the type and its module instance are subject to garbage collection in the sub-interpreter. Open questions ============== It is not immediately obvious how extensions should be handled that want to register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case (or could construct them from the module spec of the current module), although not the import file path. A C-API could be provided to register new module types in the current interpreter, given a user provided ModuleSpec. There is no inherent requirement for the module creation function to actually return a type. It could return a arbitrary callable that creates a 'modulish' object when called. Should there be a type check in place that makes sure that what it returns is a type? I don't currently see a need for this. Copyright ========= This document has been placed in the public domain.

7 46

when to fix cross-version bugs?
by Ethan Furman Sept. 7, 2013

Sept. 7, 2013

I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values (otherwise, in subclasses, the str() was used instead). Should this be fixed in 3.3 as well? What guidelines determine when a bug is fixed in previous versions? -- ~Ethan~

4 4

inspect and metaclasses
by Ethan Furman Sept. 6, 2013

Sept. 6, 2013

Part of the fix for issue #18693 is to fix inspect to look in the metaclass for class attributes (http://bugs.python.org/issue18929). In inspect.py in function get_mro() we can either add the metaclass unconditionally, or only if it is not 'type'. If we add unconditionally, then help() adds the following: class A(builtins.object) | Hello and goodbye + | + | Method resolution order: + | A + | builtins.object + | builtins.type | | Methods defined here: Do we want that, or should we just add the metaclass if it is not 'type'? -- ~Ethan~

4 8

Details of "import lock" in 3.3
by Jesus Cea Sept. 6, 2013

Sept. 6, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With importlib and other recent changes, what are the current details of "import lock"?. That is, the lock/locks held when Python code does "import", specially in the case of multithreading. Is that documented anywhere? Thanks! - -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUinv7Jlgi5GaxT1NAQLX7QP/SBNuy869xSKRVNlWmAiDLpZOR9IHm77K fy+AG/aJDpk4BMcm+KNoHZyUtJJs2kndfz+OVTeU/mmlr+aXu/wA1yNB1HAjAwWO 4ZWyaN9WLrhv8xXomUlT8iqikc/9pEspgQbJAljUvMLVouixIdxga3IqtaT3FQZh wkV9v04G4OA= =P3PI -----END PGP SIGNATURE-----

2 3

Summary of Python tracker Issues
by Python tracker Sept. 6, 2013

Sept. 6, 2013

ACTIVITY SUMMARY (2013-08-30 - 2013-09-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4195 (+13) closed 26523 (+47) total 30718 (+60) Open issues with patches: 1915 Issues opened (43) ================== #16826: Don't check for PYTHONCASEOK if interpreter started with -E http://bugs.python.org/issue16826 reopened by meador.inge #18709: SSL module fails to handle NULL bytes inside subjectAltNames g http://bugs.python.org/issue18709 reopened by neologix #18886: BytesGenerator does not handle 'binary' CTE correctly http://bugs.python.org/issue18886 opened by r.david.murray #18887: test_multiprocessing.test_connection failure http://bugs.python.org/issue18887 opened by neologix #18890: Add a raw_data_manager content manager to the email package. http://bugs.python.org/issue18890 opened by r.david.murray #18891: Master patch for content manager addtion to email package. http://bugs.python.org/issue18891 opened by r.david.murray #18893: invalid exception handling in Lib/ctypes/macholib/dyld.py http://bugs.python.org/issue18893 opened by scoder #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 opened by py.user #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 opened by py.user #18896: Remove namedtuple 255 arguments restriction http://bugs.python.org/issue18896 opened by valorien #18898: Apply the setobject optimizations to dictionaries http://bugs.python.org/issue18898 opened by rhettinger #18899: make pystone.py Py3 compatible in benchmark suite http://bugs.python.org/issue18899 opened by scoder #18900: Add the random.distrib module http://bugs.python.org/issue18900 opened by serhiy.storchaka #18902: Make ElementTree event handling more modular to allow custom t http://bugs.python.org/issue18902 opened by eli.bendersky #18903: IDLE file-completion is case-sensitive in Windows http://bugs.python.org/issue18903 opened by anikom15 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 opened by vajrasky #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 opened by Wieland.Hoffmann #18906: Create a way to always run tests in subprocesses within regrte http://bugs.python.org/issue18906 opened by eli.bendersky #18907: urllib2.open FTP open times out at 20 secs despite timeout par http://bugs.python.org/issue18907 opened by nagle #18908: Enum docs: sections leak out http://bugs.python.org/issue18908 opened by elazar #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 opened by philwebster #18911: minidom does not encode correctly when calling Document.writex http://bugs.python.org/issue18911 opened by brianvanderburg2 #18913: ssl._ssl._test_decode_cert seems to leak memory with certain c http://bugs.python.org/issue18913 opened by sYnfo #18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings http://bugs.python.org/issue18915 opened by mpb #18916: Various out-of-date Lock text in 3.2+ http://bugs.python.org/issue18916 opened by tim.peters #18917: python won't display greek characters in apache under windows http://bugs.python.org/issue18917 opened by nickl1 #18918: help('FILES') finds no documentation http://bugs.python.org/issue18918 opened by st.sempert(a)gmail.com #18919: Unify audio modules tests http://bugs.python.org/issue18919 opened by serhiy.storchaka #18921: In imaplib, cached capabilities may be out of date after login http://bugs.python.org/issue18921 opened by sjmurdoch #18923: Use the new selectors module in the subprocess module http://bugs.python.org/issue18923 opened by haypo #18925: select.poll.modify is not documented http://bugs.python.org/issue18925 opened by giampaolo.rodola #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 opened by tim.peters #18929: inspect.classify_class_attrs ignores metaclass http://bugs.python.org/issue18929 opened by ethan.furman #18930: os.spawnXX functions terminates process if second argument is http://bugs.python.org/issue18930 opened by SSchukat #18931: new selectors module should support devpoll on Solaris http://bugs.python.org/issue18931 opened by giampaolo.rodola #18932: selectors and modify() http://bugs.python.org/issue18932 opened by giampaolo.rodola #18934: multiprocessing: use selectors module http://bugs.python.org/issue18934 opened by neologix #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 opened by neologix #18936: getopt chokes on unicode option names http://bugs.python.org/issue18936 opened by jason.coombs #18937: add unittest assertion for logging http://bugs.python.org/issue18937 opened by pitrou #18943: argparse: default args in mutually exclusive groups http://bugs.python.org/issue18943 opened by arigo #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 opened by arigo #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 opened by vlad Most recent 15 issues with no replies (15) ========================================== #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 #18943: argparse: default args in mutually exclusive groups http://bugs.python.org/issue18943 #18936: getopt chokes on unicode option names http://bugs.python.org/issue18936 #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 #18932: selectors and modify() http://bugs.python.org/issue18932 #18930: os.spawnXX functions terminates process if second argument is http://bugs.python.org/issue18930 #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 #18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings http://bugs.python.org/issue18915 #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 #18903: IDLE file-completion is case-sensitive in Windows http://bugs.python.org/issue18903 #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 Most recent 15 issues waiting for review (15) ============================================= #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 #18934: multiprocessing: use selectors module http://bugs.python.org/issue18934 #18931: new selectors module should support devpoll on Solaris http://bugs.python.org/issue18931 #18919: Unify audio modules tests http://bugs.python.org/issue18919 #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 #18908: Enum docs: sections leak out http://bugs.python.org/issue18908 #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 #18899: make pystone.py Py3 compatible in benchmark suite http://bugs.python.org/issue18899 #18898: Apply the setobject optimizations to dictionaries http://bugs.python.org/issue18898 #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 #18893: invalid exception handling in Lib/ctypes/macholib/dyld.py http://bugs.python.org/issue18893 #18891: Master patch for content manager addtion to email package. http://bugs.python.org/issue18891 Top 10 most discussed issues (10) ================================= #18808: Thread.join returns before PyThreadState is destroyed http://bugs.python.org/issue18808 11 msgs #18844: allow weights in random.choice http://bugs.python.org/issue18844 11 msgs #18843: Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at P http://bugs.python.org/issue18843 10 msgs #18693: help() not helpful with enum http://bugs.python.org/issue18693 9 msgs #18726: json functions have too many positional parameters http://bugs.python.org/issue18726 9 msgs #18913: ssl._ssl._test_decode_cert seems to leak memory with certain c http://bugs.python.org/issue18913 8 msgs #18831: importlib.import_module() bypasses builtins.__import__ http://bugs.python.org/issue18831 7 msgs #18709: SSL module fails to handle NULL bytes inside subjectAltNames g http://bugs.python.org/issue18709 6 msgs #18840: Tutorial recommends pickle module without any warning of insec http://bugs.python.org/issue18840 6 msgs #18876: Problems with files opened in append mode with io module http://bugs.python.org/issue18876 6 msgs Issues closed (43) ================== #12037: test_email failures under desktop Windows http://bugs.python.org/issue12037 closed by terry.reedy #15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear http://bugs.python.org/issue15350 closed by orsenthil #16853: add a Selector to the select module http://bugs.python.org/issue16853 closed by pitrou #17224: can not open idle in python 2.7.3 http://bugs.python.org/issue17224 closed by terry.reedy #17930: Search not needed in combinations_with_replacement http://bugs.python.org/issue17930 closed by tim.peters #18105: ElementTree writes invalid files when UTF-16 encoding is speci http://bugs.python.org/issue18105 closed by eli.bendersky #18418: Thread.isAlive() sometimes True after fork http://bugs.python.org/issue18418 closed by neologix #18489: IDLE Unit test for SearchEngine.py http://bugs.python.org/issue18489 closed by terry.reedy #18672: Fix format specifiers for debug output in _sre.c http://bugs.python.org/issue18672 closed by serhiy.storchaka #18720: Switch suitable constants in the socket module to IntEnum http://bugs.python.org/issue18720 closed by python-dev #18738: String formatting (% and str.format) issues with Enum http://bugs.python.org/issue18738 closed by python-dev #18745: Test enum in test_json is ignorant of infinity value http://bugs.python.org/issue18745 closed by python-dev #18750: '' % [1] doesn't fail http://bugs.python.org/issue18750 closed by asvetlov #18756: os.urandom() fails under high load http://bugs.python.org/issue18756 closed by pitrou #18780: SystemError when formatting int subclass http://bugs.python.org/issue18780 closed by python-dev #18826: reversed() requires a sequence - Could work on any iterator? http://bugs.python.org/issue18826 closed by rhettinger #18830: Remove duplicates from a result of getclasstree() http://bugs.python.org/issue18830 closed by serhiy.storchaka #18845: 2.7.5-r2: Fatal Python error: Segmentation fault http://bugs.python.org/issue18845 closed by tim.peters #18849: Failure to try another name for tempfile when directory with c http://bugs.python.org/issue18849 closed by python-dev #18850: xml.etree.ElementTree accepts control chars. http://bugs.python.org/issue18850 closed by eli.bendersky #18851: subprocess's Popen closes stdout/stderr filedescriptors used i http://bugs.python.org/issue18851 closed by pitrou #18870: eval() uses latin-1 to decode str http://bugs.python.org/issue18870 closed by python-dev #18878: Add support of the 'with' statement to sunau.open. http://bugs.python.org/issue18878 closed by serhiy.storchaka #18882: Add threading.main_thread() function http://bugs.python.org/issue18882 closed by asvetlov #18888: Add stdlib support for random sampling with replacement http://bugs.python.org/issue18888 closed by mark.dickinson #18889: test_sax: multiple failures on Windows desktop http://bugs.python.org/issue18889 closed by tim.peters #18892: sqlite3, valued records not persisted, default ones are http://bugs.python.org/issue18892 closed by debewerker #18897: Illegal instruction at Python-2.7.5/Modules/_sre.c:1173 http://bugs.python.org/issue18897 closed by pitrou #18901: sunau.getparams should return a namedtuple http://bugs.python.org/issue18901 closed by serhiy.storchaka #18909: Segfaults on win-amd64 due to corrupt pointer to Tkapp_Interp http://bugs.python.org/issue18909 closed by haypo #18912: Intendation issue in example code in itertools.count documenta http://bugs.python.org/issue18912 closed by python-dev #18914: Confusing documentation in the urllib2 HOWTO http://bugs.python.org/issue18914 closed by michael.foord #18920: argparse module version action http://bugs.python.org/issue18920 closed by eli.bendersky #18922: Output versions of scripts to stdout http://bugs.python.org/issue18922 closed by serhiy.storchaka #18924: Enum members are easily replaced http://bugs.python.org/issue18924 closed by python-dev #18926: plistlib - str converted to bool http://bugs.python.org/issue18926 closed by VertigoRay #18928: Remove misleading documentation for random.shuffle http://bugs.python.org/issue18928 closed by dbenbenn #18933: Add link to source code in logging documentation http://bugs.python.org/issue18933 closed by python-dev #18938: Prepend Is Not A Word http://bugs.python.org/issue18938 closed by benjamin.peterson #18939: Venv docs regarding original python install http://bugs.python.org/issue18939 closed by python-dev #18940: TimedRotatingFileHandler and RotatingFileHandler fail to doRol http://bugs.python.org/issue18940 closed by python-dev #18941: RotatingFileHandler and TimedRotatingFileHandler do not respec http://bugs.python.org/issue18941 closed by python-dev #18942: _debugmallocstats() gibberish output on Windows http://bugs.python.org/issue18942 closed by tim.peters

1 0

SEEK_* constants in io and os
by Eli Bendersky Sept. 6, 2013

Sept. 6, 2013

Hello, I was looking at the possibility of replacing the SEEK_* constants by IntEnums, and the first thing that catches attention is that these constants are defined in both Lib/os.py and Lib/io.py; both places also recently started supporting SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE and os.SEEK_DATA). Additional data points: other modules take these constants as arguments - * mmap: directs to use os.SEEK_* * chunk and fcntk: spell out the numeric values. os seems to import io in some functions; can this be done always? If yes, we can just define the constants once and os.SEEK_* will alias io.SEEK_*? The other way (io taking from os) is also a possibility (maybe the preferred one because io already refers to os.SEEK_HOLE/DATA, at least in the documentation). Any ideas and suggestions are welcome, Eli

4 9

windows file closing race condition?
by Chris Withers Sept. 6, 2013

Sept. 6, 2013

Hi All, Continuous testing is a wonderful thing when it comes to finding weird edge case problems, like this one: http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3… File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", line 323, in __exit__ self.cleanup() File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", line 78, in cleanup rmtree(self.path) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 460, in rmtree return _rmtree_unsafe(path, onerror) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 362, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 371, in _rmtree_unsafe onerror(os.rmdir, path, sys.exc_info()) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 369, in _rmtree_unsafe os.rmdir(path) OSError: [WinError 145] The directory is not empty: 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a' I'm 99% certain my code is correct here, the only place I open files for writing in that directory is here: https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdir… So, from my perspective, I'm either looking at a bug in shutil.rmtree (which would be trying to delete a directory before deleting its content or failing to delete a file but ignoring an error) or the file object, when being used as a context manager, going through __exit__ without closing the file and releasing the handle. This happens very infrequently, the OS is Windows 7 and the filesystem is NTFS, if that helps... Any ideas? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

8 14

Add a new tracemalloc module to trace memory allocations
by Victor Stinner Sept. 5, 2013

Sept. 5, 2013

Hi, Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object. tracemalloc is different than Heapy or PySizer because it is focused on the location of a memory allocation rather that the object type or object content. I have an implementation of the module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc My proposed implementation for Python 3.4 is different: * reuse the PEP 445 to hook memory allocators * use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project, BSD license) instead of depending on the glib library. I simplified and adapted cfuhash for my usage * no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable * traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization. The (high-level) API should be reviewed and discussed. The most interesting part is to take "snapshots" and compare snapshots. The module can load snapshots from disk and compare them later for deeper analysis (ex: ignore some files). For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc I created the following issue to track the implementation: http://bugs.python.org/issue18874 The implementation: http://hg.python.org/features/tracemalloc * * * I also created a "pyfailmalloc" project based on the PEP 445 to inject MemoryError exceptions. I used this module to check if Python handles correctly memory allocation failures. The answer is no, I fixed many bugs (see issue #18408). Project homepage: https://bitbucket.org/haypo/pyfailmalloc Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?" What is the best place for such module? Add it to Modules/ directory but as a private module: "_failmalloc"? * * * Example of tracemalloc output (it is more readable with colors, try in a terminal). The first top is sorted by total size, whereas the second top is sorted (automatically) with the size difference. You can see for example that the linecache module likes caching data (1.5 MB after 10 seconds of tests). $ PYTHONTRACEMALLOC=1 ./python -m test ... == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow little-endian == /home/haypo/prog/python/tracemalloc/build/test_python_11087 ... [ 1/380] test_grammar [ 2/380] test_opcodes [ 3/380] test_dict [ 4/380] test_builtin [ 5/380] test_exceptions [ 6/380] test_types [ 7/380] test_unittest 2013-08-29 02:06:22: Top 25 allocations per file and line #1: <frozen importlib._bootstrap>:704: size=5 MiB, count=56227, average=105 B #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, average=118 B #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB #6: <frozen importlib._bootstrap>:274: size=703 KiB, count=4604, average=156 B #7: ???:?: size=511 KiB, count=4445, average=117 B #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, average=113 B #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, average=159 B #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=672, average=156 B #16: <frozen importlib._bootstrap>:1430: size=91 KiB, count=1193, average=78 B #17: .../tracemalloc/Lib/inspect.py:1399: size=79 KiB, count=104, average=784 B #18: .../tracemalloc/Lib/abc.py:133: size=77 KiB, count=275, average=289 B #19: .../Lib/unittest/case.py:43: size=73 KiB, count=593, average=127 B #20: .../Lib/unittest/mock.py:491: size=67 KiB, count=153, average=450 B #21: <frozen importlib._bootstrap>:1438: size=64 KiB, count=20, average=3321 B #22: .../Lib/unittest/case.py:535: size=56 KiB, count=76, average=766 B #23: .../tracemalloc/Lib/sre_compile.py:508: size=54 KiB, count=115, average=485 B #24: .../Lib/unittest/case.py:300: size=48 KiB, count=616, average=80 B #25: .../Lib/test/regrtest.py:1207: size=48 KiB, count=2, average=24 KiB 7333 more: size=4991 KiB, count=28051, average=182 B Total Python memory: size=17 MiB, count=136358, average=136 B Total process memory: size=42 MiB (ignore tracemalloc: 22 KiB) [ 8/380] test_doctest [ 9/380] test_doctest2 [ 10/380] test_support [ 11/380] test___all__ 2013-08-29 02:08:44: Top 25 allocations per file and line (compared to 2013-08-29 02:08:39) #1: <frozen importlib._bootstrap>:704: size=8 MiB (+2853 KiB), count=80879 (+24652), average=109 B #2: .../tracemalloc/Lib/linecache.py:127: size=1562 KiB (+557 KiB), count=13669 (+4964), average=117 B #3: <frozen importlib._bootstrap>:274: size=955 KiB (+252 KiB), count=6415 (+1811), average=152 B #4: .../Lib/collections/__init__.py:368: size=333 KiB (+93 KiB), count=3136 (+966), average=108 B #5: .../tracemalloc/Lib/abc.py:133: size=148 KiB (+71 KiB), count=483 (+211), average=314 B #6: .../Lib/urllib/parse.py:476: size=71 KiB (+71 KiB), count=969 (+969), average=75 B #7: .../tracemalloc/Lib/base64.py:143: size=59 KiB (+59 KiB), count=1025 (+1025), average=59 B #8: .../tracemalloc/Lib/doctest.py:1283: size=56 KiB (+56 KiB), count=507 (+507), average=113 B #9: .../tracemalloc/Lib/sre_compile.py:508: size=89 KiB (+36 KiB), count=199 (+86), average=460 B #10: <frozen importlib._bootstrap>:53: size=67 KiB (+32 KiB), count=505 (+242), average=136 B #11: <frozen importlib._bootstrap>:1048: size=61 KiB (+27 KiB), count=332 (+138), average=188 B #12: .../Lib/unittest/case.py:496: size=351 KiB (+25 KiB), count=688 (+48), average=522 B #13: .../tracemalloc/Lib/_weakrefset.py:38: size=55 KiB (+24 KiB), count=521 (+236), average=109 B #14: .../Lib/email/quoprimime.py:56: size=24 KiB (+24 KiB), count=190 (+190), average=132 B #15: .../Lib/email/quoprimime.py:57: size=24 KiB (+24 KiB), count=1 (+1) #16: .../test/support/__init__.py:1055: size=24 KiB (+24 KiB), count=1 (+1) #17: .../Lib/test/test___all__.py:48: size=23 KiB (+23 KiB), count=283 (+283), average=84 B #18: .../tracemalloc/Lib/_weakrefset.py:37: size=60 KiB (+22 KiB), count=438 (+174), average=140 B #19: .../tracemalloc/Lib/sre_parse.py:73: size=48 KiB (+20 KiB), count=263 (+114), average=189 B #20: <string>:5: size=61 KiB (+18 KiB), count=173 (+57), average=364 B #21: .../Lib/test/test___all__.py:34: size=16 KiB (+16 KiB), count=164 (+164), average=104 B #22: .../Lib/unittest/mock.py:491: size=79 KiB (+16 KiB), count=145 (+3), average=560 B #23: .../Lib/collections/__init__.py:362: size=50 KiB (+16 KiB), count=23 (+7), average=2255 B #24: .../Lib/test/test___all__.py:59: size=13 KiB (+13 KiB), count=165 (+165), average=84 B #25: .../tracemalloc/Lib/doctest.py:1291: size=13 KiB (+13 KiB), count=170 (+170), average=81 B 10788 more: size=7 MiB (-830 KiB), count=36291 (-16379), average=220 B Total Python memory: size=20 MiB (+3567 KiB), count=147635 (+20805), average=143 B Total process memory: size=49 MiB (+7 MiB) (ignore tracemalloc: 1669 KiB) Victor

6 10

Completing the email6 API changes.
by R. David Murray Sept. 3, 2013

Sept. 3, 2013

If you've read my blog (eg: on planet python), you will be aware that I dedicated August to full time email package development. At the beginning of the month I worked out a design proposal for the remaining API additions to the email package, dealing with handling message bodies in a more natural way. I posted this to the email-sig, and got...well, no objections. Barry Warsaw did review it, and told me he had no issues with the overall design, but also had no time for a detailed review. Since one way to see if a design holds together is to document and code it, I decided to go ahead and do so. This resulted in a number of small tweaks, but no major changes. I have at this point completed the coding. You can view the whole patch at: http://bugs.python.org/issue18891 which also links to three layered patches that I posted as I went along, if you prefer somewhat smaller patches. I think it would be great if I could check this in for alpha2. Since it is going in as an addition to the existing provisional code, the level of review required is not as high as for non-provisional code, I think. But I would certainly appreciate review from anyone so moved, since I haven't gotten any yet. Of course, if there is serious bikeshedding about the API, I won't make alpha2, but that's fine. The longer term goal, by the way, is to move all of this out of provisional status for 3.5. This code finishes the planned API additions for the email package to bring it fully into the world of Python3 and unicode. It does not "fix" the deep internals, which could be a future development direction (but probably only after the "old" API has been retired, which will take a while). But it does make it so that you can use the email package without having to be a MIME expert. (You can't get away with *no* MIME knowledge, but you no longer have to fuss with the details of the syntax.) To give you the flavor of how the entire new provisional API plays together, here's how you can build a complete message in your application: from email.message import MIMEMessage from email.headerregistry import Address fullmsg = MIMEMessage() fullmsg['To'] = Address('Foö Bar', 'fbar(a)example.com') fullmsg['From'] = "mè <me(a)example.com>" fullmsg['Subject'] = "j'ai un problème de python." fullmsg.set_content("et la il est monté sur moi et il commence" " a m'étouffer.") htmlmsg = MIMEMessage() htmlmsg.set_content("et la il est monté sur moi et il commence" " a m'étouffer.<img src='image1' />", subtype='html') with open('python.jpg', 'rb') as python: htmlmsg.add_related(python.read(), 'image', 'jpg', cid='image1' disposition='inline') fullmsg.make_alternative() fullmsg.attach(htmlmsg) with open('police-report.txt') as report: fullmsg.add_attachment(report.read(), filename='pölice-report.txt', params=dict(wrap='flow'), headers=( 'X-Secret-Level: top', 'X-Authorization: Monty')) Which results in: >>> for line in bytes(fullmsg).splitlines(): >>> print(line) b'To: =?utf-8?q?Fo=C3=B6?= Bar <fbar(a)example.com>' b'From: =?utf-8?q?m=C3=A8?= <me(a)example.com>' b"Subject: j'ai un =?utf-8?q?probl=C3=A8me?= de python." b'MIME-Version: 1.0' b'Content-Type: multipart/mixed; boundary="===============1710006838=="' b'' b'--===============1710006838==' b'Content-Type: multipart/alternative; boundary="===============1811969196=="' b'' b'--===============1811969196==' b'Content-Type: text/plain; charset="utf-8"' b'Content-Transfer-Encoding: 8bit' b'' b"et la il est mont\xc3\xa9 sur moi et il commence a m'\xc3\xa9touffer." b'' b'--===============1811969196==' b'MIME-Version: 1.0' b'Content-Type: multipart/related; boundary="===============1469657937=="' b'' b'--===============1469657937==' b'Content-Type: text/html; charset="utf-8"' b'Content-Transfer-Encoding: quoted-printable' b'' b"et la il est mont=C3=A9 sur moi et il commence a m'=C3=A9touffer.<img =" b"src=3D'image1' />" b'' b'--===============1469657937==' b'MIME-Version: 1.0' b'Content-Type: image/jpg' b'Content-Transfer-Encoding: base64' b'Content-Disposition: inline' b'Content-ID: image1' b'' b'ZmFrZSBpbWFnZSBkYXRhCg==' b'' b'--===============1469657937==--' b'--===============1811969196==--' b'--===============1710006838==' b'MIME-Version: 1.0' b'X-Secret-Level: top' b'X-Authorization: Monty' b'Content-Transfer-Encoding: 7bit' b'Content-Disposition: attachment; filename*=utf-8''p%C3%B6lice-report.txt" b'Content-Type: text/plain; charset="utf-8"; wrap="flow"' b'' b'il est sorti de son vivarium.' b'' b'--===============1710006838==--' If you've used the email package enough to be annoyed by it, you may notice that there are some nice things going on there, such as using CTE 8bit for the text part by default, and quoted-printable instead of base64 for utf8 when the lines are long enough to need wrapping. (Hmm. Looking at that I see I didn't fully fix a bug I had meant to fix: some of the parts have a MIME-Version header that don't need it.) All input strings are unicode, and the library takes care of doing whatever encoding is required. When you pull data out of a parsed message, you get unicode, without having to worry about how to decode it yourself. On the parsing side, after the above message has been parsed into a message object, we can do: >>> print(fullmsg['to'], fullmsg['from']) Foö Bar <"fbar(a)example.com"> mè <me(a)example.com> >>> print(fullmsg['subject']) j'ai un problème de python. >>> print(fullmsg['to'].addresses[0].display_name) Foö Bar >>> print(fullmsg.get_body(('plain',)).get_content()) et la il est monté sur moi et il commence a m'étouffer. >>> for part in fullmsg.get_body().iter_parts(): ... print(part.get_content()) et la il est monté sur moi et il commence a m'étouffer.<img src='image1' /> b'fake image data\n' >>> for attachment in fullmsg.iter_attachments(): ... print(attachment.get_content()) ... print(attachment['Content-Type'].params()) il est sorti de son vivarium. {'charset': 'utf-8', 'wrap': 'flow'} Of course, in a real program you'd actually be checking the mime types via get_content_type() and friends before getting the content and doing anything with it. Please read the new contentmanager module docs in the patch for full details of the content management part of the above API (and the headerregistry docs if you want to review the (new in 3.3) header parsing part of the above API). Feedback welcome, here or on the issue. --David PS: python jokes courtesy of someone doing a drive-by on #python-dev the other day.

4 19

Python 2.6 to end support with 2.6.9 in October 2013
by Barry Warsaw Sept. 3, 2013

Sept. 3, 2013

Hello Pythonistas, Python 2.6.9 is the last planned release of the 2.6.x series. This will be a security-only source-only release. It is currently scheduled for October 2013, and after this Python 2.6 will have reached its end-of-life and the branch will be retired. http://www.python.org/dev/peps/pep-0361/ I would like to release 2.6.9rc1 on Monday, September 30th, and 2.6.9 final on Monday, October 28th. I've added both dates to the Python calendar. Here are the list of candidates still to be fixed for 2.6.9: - 18747 - Re-seed OpenSSL's PRNG after fork - 16037 - httplib: header parsing is not delimited - 16038 - ftplib: unlimited readline() from connection - 16039 - imaplib: unlimited readline() from connection - 16040 - nntplib: unlimited readline() from connection - 16041 - poplib: unlimited readline() from connection - 16042 - smtplib: unlimited readline() from connection - 16043 - xmlrpc: gzip_decode has unlimited read() These were the ones I previously had on my list, and I've now marked these all as release blockers for 2.6.9... for now. If you know of any others that I should be aware of, please let me know. If you can contribute to 2.6.9 by reviewing, testing, developing, or commenting, that would be greatly appreciated. I will be spending some time triaging these and any other issues that get identified as possible 2.6.9 candidates. If you have any questions regarding 2.6.9, please contact me via mailing list or IRC. Cheers, -Barry

2 1