Mailman 3 June 2013 - Python-Dev

PEP 445 delegate
by Antoine Pitrou 23 Jun '13

23 Jun '13

Hello, I've been appointed PEP 445 delegate by Nick and Guido. I would like to know if there are still pending changes to the PEP. If not, I expect to give it a review in the coming days or weeks, and then make a final pronouncement (which will probably be positive anyway). Regards Antoine.

2 2

stat module in C -- what to do with stat.py?
by Christian Heimes 22 Jun '13

22 Jun '13

Hello, I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values. For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily. What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy. 1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py Opinions? Christian [1] http://bugs.python.org/issue11016

23 39

[python 2.7] a question about exporting a new method to socket object
by Ani Sinha 21 Jun '13

21 Jun '13

Hi python devs : As a part of exploring an idea, I am trying to add a new method to the python socket object. Here's what I have been trying to do (the patch may be whitespace damaged so please bear with me) : Index: Python-2.7/Modules/socketmodule.c =================================================================== --- Python-2.7.orig/Modules/socketmodule.c +++ Python-2.7/Modules/socketmodule.c @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a recv(buflen[, flags]) -- receive data\n\ recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\ recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\ +arecvfrom(buflen[, flags]) -- same as recvfrom \n\ recvfrom_into(buffer[, nbytes, [, flags])\n\ -- receive data and sender\'s address (into a buffer)\n\ sendall(data[, flags]) -- send all data\n\ @@ -468,6 +469,13 @@ static PyTypeObject sock_type; #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <= 0.0) #endif @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc, \n\ Like recv(buffersize, flags) but also return the sender's address info."); +static PyObject * +sock_arecvfrom(PySocketSockObject *s, PyObject *args) +{ + return sock_recvfrom(s,args); +} + +PyDoc_STRVAR(arecvfrom_doc, +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\ +\n\ + experimental stuff"); /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */ @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = { recv_into_doc}, {"recvfrom", (PyCFunction)sock_recvfrom, METH_VARARGS, recvfrom_doc}, + {"arecvfrom", (PyCFunction)sock_arecvfrom, METH_VARARGS, + arecvfrom_doc}, {"recvfrom_into", (PyCFunction)sock_recvfrom_into, METH_VARARGS | METH_KEYWORDS, recvfrom_into_doc}, {"send", (PyCFunction)sock_send, METH_VARARGS, When I compile this and run a simple script like the following that uses arecvfrom() : sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW ) sock.bind( ( intf, ETH_P_ALL ) ) (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT ) I get this exception : AttributeError: '_socketobject' object has no attribute 'arecvfrom' So what am I doing wrong? How do I export this new socket method? any help/pointer will be greatly appreciated. cheers, ani -- Ani

3 3

Summary of Python tracker Issues
by Python tracker 21 Jun '13

21 Jun '13

ACTIVITY SUMMARY (2013-06-14 - 2013-06-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4057 (+15) closed 25993 (+47) total 30050 (+62) Open issues with patches: 1804 Issues opened (42) ================== #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 reopened by serhiy.storchaka #18216: gettext doesn't check MO versions http://bugs.python.org/issue18216 opened by jwilk #18217: Deprecate and remove gettext.install http://bugs.python.org/issue18217 opened by alex #18219: csv.DictWriter is slow when writing files with large number of http://bugs.python.org/issue18219 opened by mtraskin #18220: In itertools.islice() make prototype like in help() http://bugs.python.org/issue18220 opened by py.user #18221: abspath strips trailing spaces on win32 http://bugs.python.org/issue18221 opened by Jeremy.Gray #18222: os.path.abspath should accept multiple path parts and join the http://bugs.python.org/issue18222 opened by Łukasz.Balcerzak #18224: pyvenv pydoc.py script causing AttributeErrors on Windows http://bugs.python.org/issue18224 opened by peter(a)psantoro.net #18226: IDLE Unit test for FormatParagrah.py http://bugs.python.org/issue18226 opened by Todd.Rovito #18227: Use Python memory allocators in external libraries like zlib o http://bugs.python.org/issue18227 opened by haypo #18228: AIX locale parsing failure http://bugs.python.org/issue18228 opened by David.Edelsohn #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 opened by joru #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 opened by brett.cannon #18231: What's new in Python should explain what's new in UCD http://bugs.python.org/issue18231 opened by belopolsky #18232: running a suite with no tests is not an error http://bugs.python.org/issue18232 opened by rbcollins #18233: SSLSocket.getpeercertchain() http://bugs.python.org/issue18233 opened by christian.heimes #18234: Unicodedata module should provide access to codepoint aliases http://bugs.python.org/issue18234 opened by belopolsky #18235: _sysconfigdata.py wrong on AIX installations http://bugs.python.org/issue18235 opened by David.Edelsohn #18236: int() and float() do not accept strings with trailing separato http://bugs.python.org/issue18236 opened by belopolsky #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 opened by jtratner #18240: hmac unnecessarily restricts input to "bytes" http://bugs.python.org/issue18240 opened by jborgstrom #18241: Add unique option to heapq functions http://bugs.python.org/issue18241 opened by pitrou #18242: IDLE should not be replacing warnings.formatwarning http://bugs.python.org/issue18242 opened by brett.cannon #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 opened by jwilk #18244: singledispatch: When virtual-inheriting ABCs at distinct point http://bugs.python.org/issue18244 opened by ecatmur #18246: tkinter.Text() add a newline to the content - bug? http://bugs.python.org/issue18246 opened by Friedrich.Spee.von.Langenfeld #18249: Incorrect and incomplete help docs for close() method http://bugs.python.org/issue18249 opened by DaveA #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 opened by archardlias #18255: CPython setup.py problems http://bugs.python.org/issue18255 opened by ronaldoussoren #18257: Two copies of python-config http://bugs.python.org/issue18257 opened by ronaldoussoren #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 opened by zach.ware #18260: configparser: TypeError occurs when handling errors in files w http://bugs.python.org/issue18260 opened by Arfrever #18262: ZipInfo.external_attr are not documented http://bugs.python.org/issue18262 opened by techtonik #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 opened by ncoghlan #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 opened by zach.ware #18268: ElementTree.fromstring non-deterministically gives unicode tex http://bugs.python.org/issue18268 opened by Brendan.OConnor #18269: Add new parameter format for converter function w/ position nu http://bugs.python.org/issue18269 opened by techtonik #18270: IDLE on OS X fails with Attribute Error if no initial shell an http://bugs.python.org/issue18270 opened by dbackhaus #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 opened by zach.ware #18275: Make isinstance() work with super type instances http://bugs.python.org/issue18275 opened by brett.cannon #18276: posixpath.c:_fd_converter() should use PyObject_AsFileDescript http://bugs.python.org/issue18276 opened by christian.heimes #18277: Queue is empty right after put from the same process/thread http://bugs.python.org/issue18277 opened by shwouchk Most recent 15 issues with no replies (15) ========================================== #18277: Queue is empty right after put from the same process/thread http://bugs.python.org/issue18277 #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18260: configparser: TypeError occurs when handling errors in files w http://bugs.python.org/issue18260 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 #18246: tkinter.Text() add a newline to the content - bug? http://bugs.python.org/issue18246 #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 #18242: IDLE should not be replacing warnings.formatwarning http://bugs.python.org/issue18242 #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 #18236: int() and float() do not accept strings with trailing separato http://bugs.python.org/issue18236 #18232: running a suite with no tests is not an error http://bugs.python.org/issue18232 #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 #18226: IDLE Unit test for FormatParagrah.py http://bugs.python.org/issue18226 Most recent 15 issues waiting for review (15) ============================================= #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 #18255: CPython setup.py problems http://bugs.python.org/issue18255 #18244: singledispatch: When virtual-inheriting ABCs at distinct point http://bugs.python.org/issue18244 #18240: hmac unnecessarily restricts input to "bytes" http://bugs.python.org/issue18240 #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 #18233: SSLSocket.getpeercertchain() http://bugs.python.org/issue18233 #18222: os.path.abspath should accept multiple path parts and join the http://bugs.python.org/issue18222 #18220: In itertools.islice() make prototype like in help() http://bugs.python.org/issue18220 #18219: csv.DictWriter is slow when writing files with large number of http://bugs.python.org/issue18219 #18214: Stop purging modules which are garbage collected before shutdo http://bugs.python.org/issue18214 #18206: license url in site.py should always use X.Y.Z form of version http://bugs.python.org/issue18206 #18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_ http://bugs.python.org/issue18203 #18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly http://bugs.python.org/issue18191 Top 10 most discussed issues (10) ================================= #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 16 msgs #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 15 msgs #18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_ http://bugs.python.org/issue18203 9 msgs #18103: Create a GUI test framework for Idle http://bugs.python.org/issue18103 8 msgs #16499: CLI option for isolated mode http://bugs.python.org/issue16499 7 msgs #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 7 msgs #18153: python imaplib - error 'unexpected repsonse' http://bugs.python.org/issue18153 7 msgs #18162: Add index attribute to IndexError http://bugs.python.org/issue18162 7 msgs #18269: Add new parameter format for converter function w/ position nu http://bugs.python.org/issue18269 7 msgs #13483: Use VirtualAlloc to allocate memory arenas http://bugs.python.org/issue13483 6 msgs Issues closed (43) ================== #14015: surrogateescape largely missing from documentation http://bugs.python.org/issue14015 closed by python-dev #15198: multiprocessing Pipe send of non-picklable objects doesn't rai http://bugs.python.org/issue15198 closed by sbt #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 closed by sbt #17018: Inconsistent behaviour of methods waiting for child process http://bugs.python.org/issue17018 closed by sbt #17177: Deprecate imp http://bugs.python.org/issue17177 closed by brett.cannon #17689: Fix test discovery for test_tarfile.py http://bugs.python.org/issue17689 closed by zach.ware #17907: Deprecate imp.new_module() in favour of types.ModuleType http://bugs.python.org/issue17907 closed by brett.cannon #17961: Use enum names as values in enum.Enum() functional API http://bugs.python.org/issue17961 closed by ncoghlan #18014: Problem compiling on Windows, VC++Express 2010 http://bugs.python.org/issue18014 closed by terry.reedy #18057: Register NamespaceLoader with importlib.abc.Loader http://bugs.python.org/issue18057 closed by brett.cannon #18058: Define is_package for NamespaceLoader http://bugs.python.org/issue18058 closed by brett.cannon #18076: Implement importlib.util.decode_source() http://bugs.python.org/issue18076 closed by brett.cannon #18115: Use importlib.util.module_to_load in all loaders in importlib http://bugs.python.org/issue18115 closed by brett.cannon #18147: SSL: diagnostic functions to list loaded CA certs http://bugs.python.org/issue18147 closed by christian.heimes #18149: filecmp.cmp() incorrect results when previously compared file http://bugs.python.org/issue18149 closed by ned.deily #18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n http://bugs.python.org/issue18167 closed by serhiy.storchaka #18192: Move imp.get_magic() to importlib http://bugs.python.org/issue18192 closed by brett.cannon #18193: Move imp.reload() to importlib http://bugs.python.org/issue18193 closed by brett.cannon #18194: Move imp.source_from_cache/cache_from_source to importlib http://bugs.python.org/issue18194 closed by brett.cannon #18197: insufficient error checking causes crash on windows http://bugs.python.org/issue18197 closed by ncoghlan #18202: Minor fixes for test_coding http://bugs.python.org/issue18202 closed by serhiy.storchaka #18207: OpenSSL may ignore seconds in notAfter http://bugs.python.org/issue18207 closed by christian.heimes #18218: In itertools.count() clarify the starting point http://bugs.python.org/issue18218 closed by python-dev #18223: Refactor test_tarfile http://bugs.python.org/issue18223 closed by serhiy.storchaka #18225: ctypes Structure data size is incorrect http://bugs.python.org/issue18225 closed by alex #18238: test_signal.py wait_helper hangs on AIX http://bugs.python.org/issue18238 closed by haypo #18239: In itertools docstring update arguments in count() example http://bugs.python.org/issue18239 closed by python-dev #18245: In itertools.groupby() make data plural http://bugs.python.org/issue18245 closed by r.david.murray #18247: Add Lib/test/data/ to .gitignore http://bugs.python.org/issue18247 closed by python-dev #18248: fficonfig.py.in wrong for AIX http://bugs.python.org/issue18248 closed by python-dev #18250: In itertools.repeat() object shadows object() http://bugs.python.org/issue18250 closed by akuchling #18251: test_subprocess_jy fails when Argument has embedded quote http://bugs.python.org/issue18251 closed by r.david.murray #18252: timeit makes code run faster? http://bugs.python.org/issue18252 closed by r.david.murray #18253: make standard library PEP8 compliant http://bugs.python.org/issue18253 closed by r.david.murray #18256: thread_pthread.h fixes for AIX http://bugs.python.org/issue18256 closed by pitrou #18259: Declare sethostname in socketmodule.c for AIX http://bugs.python.org/issue18259 closed by christian.heimes #18261: Confusing description in Minimal DOM implementation http://bugs.python.org/issue18261 closed by ned.deily #18263: python.man: no documentation for -b, -X http://bugs.python.org/issue18263 closed by orsenthil #18265: typedef lookupfunc defined by not used http://bugs.python.org/issue18265 closed by David.Edelsohn #18267: xmlrpc.client documentation multicall example missleading for http://bugs.python.org/issue18267 closed by python-dev #18271: get_payload method returns bytes which cannot be decoded using http://bugs.python.org/issue18271 closed by r.david.murray #18272: In itertools recipes there is a typo in __builtins__ http://bugs.python.org/issue18272 closed by python-dev #18274: python: not found http://bugs.python.org/issue18274 closed by brett.cannon

1 0

PEP 409 and the stdlib
by Ethan Furman 21 Jun '13

21 Jun '13

As a quick reminder, PEP 409 allows this: try: ... except AnError: raise SomeOtherError from None so that if the exception is not caught, we get the traditional single exception traceback, instead of the new: During handling of the above exception, another exception occurred My question: How do we go about putting this in the stdlib? Is this one of the occasions where we don't do it unless we're modifying a module already for some other reason? For that matter, should we? Pros: Makes tracebacks much less confusing, especially coming from a library Cons: Could hide bugs unrelated to what is being caught and transformed -- ~Ethan~

12 32

PEP 445: Add new APIs to customize Python memory allocators (second round)
by Victor Stinner 20 Jun '13

20 Jun '13

Hi, I changed the PEP 445 according to the discussing on python-dev. Read it online: http://www.python.org/dev/peps/pep-0445/ Changes: * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or PYALLOC_PYOBJECT * rename: - PyMemBlockAllocator structure => PyMemAllocator - PyMemMappingAllocator structure => PyObjectArenaAllocator - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator() - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator() * group get/set functions to only keep 2 functions: PyMem_GetAllocator() and PyMem_SetAllocator() * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined behaviour * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same allocator * Add more references for external libraries As expected, most changes occurred in the Proposal section. Full diff: http://hg.python.org/peps/rev/a17ebebe52ca I also updated the implementation attached to: http://bugs.python.org/issue3329 Who is going give the final decision on this PEP? Guido? Another candidate? Victor

3 4

How about we extend C3 to support ABCs?
by Łukasz Langa 20 Jun '13

20 Jun '13

Review and comments on the ticket needed, please. http://bugs.python.org/issue18244#msg191535 -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

1 0

RFC: PEP 445: Add new APIs to customize Python memory allocators
by Victor Stinner 20 Jun '13

20 Jun '13

If you prefer the HTML version: http://www.python.org/dev/peps/pep-0445/ PEP: 445 Title: Add new APIs to customize Python memory allocators Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-june-2013 Python-Version: 3.4 Abstract ======== Add new APIs to customize Python memory allocators. Rationale ========= Use cases: * Application embedding Python may want to isolate Python memory from the memory of the application, or may want to use a different memory allocator optimized for its Python usage * Python running on embedded devices with low memory and slow CPU. A custom memory allocator may be required to use efficiently the memory and/or to be able to use all the memory of the device. * Debug tool to: - track the memory usage (memory leaks) - get the Python filename and line number where an object was allocated - detect buffer underflow, buffer overflow and detect misuse of Python allocator APIs (builtin Python debug hooks) - force allocation to fail to test handling of ``MemoryError`` exception Proposal ======== API changes ----------- * Add new GIL-free (no need to hold the GIL) memory allocator functions: - ``void* PyMem_RawMalloc(size_t size)`` - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` - ``void PyMem_RawFree(void *ptr)`` - the behaviour of requesting zero bytes is not defined: return *NULL* or a distinct non-*NULL* pointer depending on the platform. * Add a new ``PyMemBlockAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 3 functions */ void *ctx; /* allocate a memory block */ void* (*malloc) (void *ctx, size_t size); /* allocate or resize a memory block */ void* (*realloc) (void *ctx, void *ptr, size_t new_size); /* release a memory block */ void (*free) (void *ctx, void *ptr); } PyMemBlockAllocator; * Add new functions to get and set internal functions of ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()`` and ``PyMem_RawFree()``: - ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` - default allocator: ``malloc()``, ``realloc()``, ``free()`` * Add new functions to get and set internal functions of ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``: - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it would be treated as an error. - default allocator: ``malloc()``, ``realloc()``, ``free()``; ``PyMem_Malloc(0)`` calls ``malloc(1)`` and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)`` * Add new functions to get and set internal functions of ``PyObject_Malloc()``, ``PyObject_Realloc()`` and ``PyObject_Free()``: - ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it would be treated as an error. - default allocator: the *pymalloc* allocator * Add a new ``PyMemMappingAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size); /* release a memory mapping */ void (*free) (void *ctx, void *ptr, size_t size); } PyMemMappingAllocator; * Add a new function to get and set the memory mapping allocator: - ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)`` - ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)`` - Currently, this allocator is only used internally by *pymalloc* to allocate arenas. * Add a new function to setup the builtin Python debug hooks when memory allocators are replaced: - ``void PyMem_SetupDebugHooks(void)`` - the function does nothing is Python is compiled not compiled in debug mode * The following memory allocators always returns *NULL* if size is greater than ``PY_SSIZE_T_MAX`` (check before calling the internal function): ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``, ``PyMem_Malloc()``, ``PyMem_Realloc()``, ``PyObject_Malloc()``, ``PyObject_Realloc()``. The builtin Python debug hooks were introduced in Python 2.3 and implement the following checks: * Newly allocated memory is filled with the byte ``0xCB``, freed memory is filled with the byte ``0xDB``. * Detect API violations, ex: ``PyObject_Free()`` called on a memory block allocated by ``PyMem_Malloc()`` * Detect write before the start of the buffer (buffer underflow) * Detect write after the end of the buffer (buffer overflow) Other changes ------------- * ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in debug mode * ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of ``malloc()`` if size is greater or equal than ``SMALL_REQUEST_THRESHOLD`` (512 bytes), and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of ``realloc()`` * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or ``PyMem_RawMalloc()`` if the GIL is not held * Configure external libraries like zlib or OpenSSL to allocate memory using ``PyMem_RawMalloc()`` Examples ======== Use case 1: Replace Memory Allocator, keep pymalloc ---------------------------------------------------- Dummy example wasting 2 bytes per memory block, and 10 bytes per memory mapping:: #include <stdlib.h> int block_padding = 2; int mapping_padding = 10; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void* my_alloc_mapping(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void my_free_mapping(void *ctx, void *ptr, size_t size) { free(ptr); } void setup_custom_allocator(void) { PyMemBlockAllocator block; PyMemMappingAllocator mapping; block.ctx = &block_padding; block.malloc = my_malloc; block.realloc = my_realloc; block.free = my_free; PyMem_SetRawAllocator(&block); PyMem_SetAllocator(&block); mapping.ctx = &mapping_padding; mapping.alloc = my_alloc_mapping; mapping.free = my_free_mapping; PyMem_SetMappingAllocator(mapping); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator are not thread-safe. Use case 2: Replace Memory Allocator, override pymalloc -------------------------------------------------------- If your allocator is optimized for allocation of small objects (less than 512 bytes) with a short lifetime, pymalloc can be overriden (replace ``PyObject_Malloc()``). Dummy example wasting 2 bytes per memory block:: #include <stdlib.h> int padding = 2; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void setup_custom_allocator(void) { PyMemBlockAllocator alloc; alloc.ctx = &padding; alloc.malloc = my_malloc; alloc.realloc = my_realloc; alloc.free = my_free; PyMem_SetRawAllocator(&alloc); PyMem_SetAllocator(&alloc); PyObject_SetAllocator(&alloc); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator are not thread-safe. Use case 3: Setup Allocator Hooks --------------------------------- Example to setup hooks on all memory allocators:: struct { PyMemBlockAllocator raw; PyMemBlockAllocator mem; PyMemBlockAllocator obj; /* ... */ } hook; static void* hook_malloc(void *ctx, size_t size) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; /* ... */ ptr = alloc->malloc(alloc->ctx, size); /* ... */ return ptr; } static void* hook_realloc(void *ctx, void *ptr, size_t new_size) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; void *ptr2; /* ... */ ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); /* ... */ return ptr2; } static void hook_free(void *ctx, void *ptr) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; /* ... */ alloc->free(alloc->ctx, ptr); /* ... */ } void setup_hooks(void) { PyMemBlockAllocator alloc; static int installed = 0; if (installed) return; installed = 1; alloc.malloc = hook_malloc; alloc.realloc = hook_realloc; alloc.free = hook_free; PyMem_GetRawAllocator(&hook.raw); alloc.ctx = &hook.raw; PyMem_SetRawAllocator(&alloc); PyMem_GetAllocator(&hook.mem); alloc.ctx = &hook.mem; PyMem_SetAllocator(&alloc); PyObject_GetAllocator(&hook.obj); alloc.ctx = &hook.obj; PyObject_SetAllocator(&alloc); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if hooks are not thread-safe. .. note:: ``PyMem_SetupDebugHooks()`` does not need to be called: Python debug hooks are installed automatically at startup. Performances ============ Results of the `Python benchmarks suite <http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x faster, some tests are 1.04 slower, significant is between 115 and -191. Results of pybench benchmark: "+0.1%" slower globally (diff between -4.9% and +5.6%). The full reports are attached to the issue #3329. Alternatives ============ Only one get/set function for block allocators ---------------------------------------------- Replace the 6 functions: * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` with 2 functions with an additional *domain* argument: * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` These functions return 0 on success, or -1 if the domain is unknown. where domain is one of these values: * ``PYALLOC_PYMEM`` * ``PYALLOC_PYMEM_RAW`` * ``PYALLOC_PYOBJECT`` Drawback: the caller has to check if the result is 0, or handle the error. Make PyMem_Malloc() reuse PyMem_RawMalloc() by default ------------------------------------------------------ ``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So calling ``PyMem_SetRawAllocator()`` would also also patch ``PyMem_Malloc()`` indirectly. .. note:: In the implementation of this PEP (issue #3329), ``PyMem_RawMalloc(0)`` calls ``malloc(0)``, whereas ``PyMem_Malloc(0)`` calls ``malloc(1)``. Add a new PYDEBUGMALLOC environment variable -------------------------------------------- To be able to use the Python builtin debug hooks even when a custom memory allocator replaces the default Python allocator, an environment variable ``PYDEBUGMALLOC`` can be added to setup these debug function hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``. If the environment variable is present, ``PyMem_SetRawAllocator()``, ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall automatically the hook on top of the new allocator. An new environment variable would make the Python initialization even more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_ tries to simply the CPython startup sequence. Use macros to get customizable allocators ----------------------------------------- To have no overhead in the default configuration, customizable allocators would be an optional feature enabled by a configuration option or by macros. Not having to recompile Python makes debug hooks easier to use in practice. Extensions modules don't have to be recompiled with macros. Pass the C filename and line number ----------------------------------- Define allocator functions as macros using ``__FILE__`` and ``__LINE__`` to get the C filename and line number of a memory allocation. Example of ``PyMem_Malloc`` macro with the modified ``PyMemBlockAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 3 functions */ void *ctx; /* allocate a memory block */ void* (*malloc) (void *ctx, const char *filename, int lineno, size_t size); /* allocate or resize a memory block */ void* (*realloc) (void *ctx, const char *filename, int lineno, void *ptr, size_t new_size); /* release a memory block */ void (*free) (void *ctx, const char *filename, int lineno, void *ptr); } PyMemBlockAllocator; void* _PyMem_MallocTrace(const char *filename, int lineno, size_t size); /* need also a function for the Python stable ABI */ void* PyMem_Malloc(size_t size); #define PyMem_Malloc(size) \ _PyMem_MallocTrace(__FILE__, __LINE__, size) Passing a filename and a line number to each allocator makes the API more complex: pass 3 new arguments, instead of just a context argument, to each allocator function. The GC allocator functions should also be patched. For example, ``_PyObject_GC_Malloc()`` is used in many C functions and so objects of differenet types would have the same allocation location. Such changes add too much complexity for a little gain. GIL-free PyMem_Malloc() ----------------------- When Python is compiled in debug mode, ``PyMem_Malloc()`` calls indirectly ``PyObject_Malloc()`` which requires the GIL to be held. That's why ``PyMem_Malloc()`` must be called with the GIL held. This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call ``malloc()``. So the "GIL must be held" restriction may be removed from ``PyMem_Malloc()``. Allowing to call ``PyMem_Malloc()`` without holding the GIL might break applications which setup their own allocators or allocator hooks. Holding the GIL is convinient to develop a custom allocator: no need to care of other threads. It is also convinient for a debug allocator hook: Python internal objects can be safetly inspected. Calling ``PyGILState_Ensure()`` in a memory allocator may have unexpected behaviour, especially at Python startup and at creation of a new Python thread state. Don't add PyMem_RawMalloc() --------------------------- Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is held. Otherwise, keep ``malloc()`` unchanged. The ``PyMem_Malloc()`` is used without the GIL held in some Python functions. For example, the ``main()`` and ``Py_Main()`` functions of Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or ``PyMem_RawMalloc()``). If an hook is used to the track memory usage, the ``malloc()`` memory will not be seen. Remaining ``malloc()`` may allocate a lot of memory and so would be missed in reports. Use existing debug tools to analyze the memory ---------------------------------------------- There are many existing debug tools to analyze the memory. Some examples: `Valgrind <http://valgrind.org/>`_, `Purify <http://ibm.com/software/awdtools/purify/>`_, `Clang AddressSanitizer <http://code.google.com/p/address-sanitizer/>`_, `failmalloc <http://www.nongnu.org/failmalloc/>`_, etc. The problem is to retrieve the Python object related to a memory pointer to read its type and/or content. Another issue is to retrieve the location of the memory allocation: the C backtrace is usually useless (same reasoning than macros using ``__FILE__`` and ``__LINE__``), the Python filename and line number (or even the Python traceback) is more useful. Classic tools are unable to introspect Python internals to collect such information. Being able to setup a hook on allocators called with the GIL held allow to collect a lot of useful data from Python internals. Add msize() ----------- Add another field to ``PyMemBlockAllocator`` and ``PyMemMappingAllocator``:: size_t msize(void *ptr); This function returns the size of a memory block or a memory mapping. Return (size_t)-1 if the function is not implemented or if the pointer is unknown (ex: NULL pointer). On Windows, this function can be implemented using ``_msize()`` and ``VirtualQuery()``. No context argument ------------------- Simplify the signature of allocator functions, remove the context argument: * ``void* malloc(size_t size)`` * ``void* realloc(void *ptr, size_t new_size)`` * ``void free(void *ptr)`` It is likely for an allocator hook to be reused for ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even ``PyMem_SetRawAllocator()``, but the hook must call a different function depending on the allocator. The context is a convenient way to reuse the same custom allocator or hook for different Python allocators. External libraries ================== Python should try to reuse the same prototypes for allocator functions than other libraries. Libraries used by Python: * OpenSSL: `CRYPTO_set_mem_functions() <http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984f…>`_ to set memory management functions globally * expat: `parserCreate() <http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l724>`_ has a per-instance memory handler Other libraries: * glib: `g_mem_set_vtable() <http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-…>`_ * libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_, global See also the `GNU libc: Memory Allocation Hooks <http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_. Memory allocators ================= The C standard library provides the well known ``malloc()`` function. Its implementation depends on the platform and of the C library. The GNU C library uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD uses `jemalloc <http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which is part of `gperftools <http://code.google.com/p/gperftools/>`_. ``malloc()`` uses two kinds of memory: heap and memory mappings. Memory mappings are usually used for large allocations (ex: larger than 256 KB), whereas the heap is used for small allocations. On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and it is contiguous. On Windows, the heap is handled by ``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by ``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be discontiguous. Releasing a memory mapping gives back immediatly the memory to the system. On UNIX, heap memory is only given back to the system if it is at the end of the heap. Otherwise, the memory will only be given back to the system when all the memory located after the released memory are also released. To allocate memory in the heap, the allocator tries to reuse free space. If there is no contiguous space big enough, the heap must be increased, even if we have more free space than required size. This issue is called the "memory fragmentation": the memory usage seen by the system may be much higher than real usage. On Windows, ``HeapAlloc()`` creates a new memory mapping with ``VirtualAlloc()`` if there is not enough free contiguous memory. CPython has a *pymalloc* allocator for allocations smaller than 512 bytes. This allocator is optimized for small objects with a short lifetime. It uses memory mappings called "arenas" with a fixed size of 256 KB. Other allocators: * Windows provides a `Low-fragmentation Heap <http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%…>`_. * The Linux kernel uses `slab allocation <http://en.wikipedia.org/wiki/Slab_allocation>`_. * The glib library has a `Memory Slice API <https://developer.gnome.org/glib/unstable/glib-Memory-Slices.html>`_: efficient way to allocate groups of equal-sized chunks of memory Links ===== CPython issues related to memory allocation: * `Issue #3329: Add new APIs to customize memory allocators <http://bugs.python.org/issue3329>`_ * `Issue #13483: Use VirtualAlloc to allocate memory arenas <http://bugs.python.org/issue13483>`_ * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe <http://bugs.python.org/issue16742>`_ * `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_ * `Issue #18227: Use Python memory allocators in external libraries like zlib or OpenSSL <http://bugs.python.org/issue18227>`_ Projects analyzing the memory usage of Python applications: * `pytracemalloc <https://pypi.python.org/pypi/pytracemalloc>`_ * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer (developed for Python 2.4) <http://pysizer.8325.org/>`_

6 20

pyparallel and new memory API discussions...
by Trent Nelson 19 Jun '13

19 Jun '13

The new memory API discussions (and PEP) warrant a quick pyparallel update: a couple of weeks after PyCon, I came up with a solution for the biggest show-stopper that has been plaguing pyparallel since its inception: being able to detect the modification of "main thread" Python objects from within a parallel context. For example, `data.append(4)` in the example below will generate an AssignmentError exception, because data is a main thread object, and `data.append(4)` gets executed from within a parallel context:: data = [ 1, 2, 3 ] def work(): data.append(4) async.submit_work(work) The solution turned out to be deceptively simple: 1. Prior to running parallel threads, lock all "main thread" memory pages as read-only (via VirtualProtect on Windows, mprotect on POSIX). 2. Detect attempts to write to main thread pages during parallel thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), and raise an exception instead (detection is done in the ceval frame exec loop). 3. Prior to returning control back to the main thread (which will be paused whilst all the parallel threads are running), unlock all the "main thread" pages. 4. Pause all parallel threads while the main thread runs. 5. Go back to 1. I got a proof-of-concept working on Windows a while back (and also played around with large page support in the same commit). The main changes were to obmalloc.c: https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe80… The key was the introduction of two new API calls, intended to be called by the pyparallel.c infrastructure: _PyMem_LockMainThreadPages() _PyMem_UnlockMainThreadPages() The implementation is pretty simple: +int +_PyMem_LockMainThreadPages(void) +{ + DWORD old = 0; + + if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, &old)) { + PyErr_SetFromWindowsErr(0); + return -1; + } Note the `base_addr` and `nbytes_committed` argument. Basically, I re-organized obmalloc.c a little bit such that we never actually call malloc() directly. Instead, we exploit the ability to reserve huge virtual address ranges without actually committing the memory, giving us a fixed `base_addr` void pointer that we can pass to calls like VirtualProtect or mprotect. We then incrementally commit more pages as demand increases, and simply adjust our `nbytes_committed` counter as we go along. The net effect is that we can call VirtualProtect/mprotect once, with a single base void pointer and size_t range, and immediately affect the protection of all memory pages that fall within that range. As an added bonus, we also get a very cheap and elegant way to test if a pointer (or any arbitrary memory address, actually) belongs to the main thread's memory range (at least in comparison to the existing _PyMem_InRange black magic). (This is very useful for my pyparallel infrastructure, which makes extensive use of conditional logic based on address tests.) (Side-bar: a side-effect of the approach I've used in the proof- of-concept (by only having a single base addr pointer) is that we effectively limit the maximum memory we could eventually commit. I actually quite like this -- in fact, I'd like to tweak it such that we can actually expose min/max memory values to the Python interpreter at startup (analogous to the JVM). Having known upper bounds on maximum memory usage will vastly simplify some other areas of my pyparallel work (like the async socket stuff). For example, consider network programs these days that take a "max clients" configuration parameter. That seems a bit backwards to me. It would be better if we simply said, "here, Python, you have 1GB to work with". That allows us to calculate how many clients we could simultaneously serve based on socket memory requirements, which allows for much more graceful behavior under load than leaving it open-ended. Maximum memory constraints would also be useful for the parallel.map(callable, iterable) stuff I've got in the works, as it'll allow us to optimally chunk work and assign to threads based on available memory.) So, Victor, I'm interested to hear how the new API you're proposing will affect this solution I've come up with for pyparallel; I'm going to be absolutely dependent upon the ability to lock main thread pages as read-only in one fell-swoop -- am I still going to be able to do that with your new API in place? Regards, Trent.

4 7

Re: [Python-Dev] [Python-checkins] cpython (3.3): ctypes: AIX needs an explicit #include <alloca.h> to get alloca()
by Jeremy Kloth 18 Jun '13

18 Jun '13

On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner <python-checkins(a)python.org> wrote: > diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c > --- a/Modules/_ctypes/callproc.c > +++ b/Modules/_ctypes/callproc.c > @@ -70,6 +70,7 @@ > > #include <ffi.h> > #include "ctypes.h" > +#include <alloca.h> This header is not present on Windows, thus breaking all the Windows buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? -- Jeremy Kloth

3 3