Mailman 3 October 2017 - Python-checkins

[65 flat] Results for Python (master branch) 2017-10-31
by lp_benchmark_robot＠intel.com 31 Oct '17

31 Oct '17

Results for project python/master, build date: 2017-10-31 03:03:14-07:00. - commit: f0f62cc - previous commit: a231428 - revision date: 2017-10-31 00:07:36-04:00 - environment: Broadwell-EP - cpu: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz 2x22 cores, stepping 1, LLC 55 MB - mem: 128 GB - os: Ubuntu 16.04.2 LTS - kernel: 4.4.0-62-generic x86_64 GNU/Linux Baseline results were generated using release v3.6.0, with hash 5c4568a from 2016-12-22 23:38:47+00:00. +-----+------------------------+--------+------------+------------+------------+ | | |relative|change since|change since|current rev | | | benchmark|std_dev*| last run | baseline |run with PGO| +-----+------------------------+--------+------------+------------+------------+ | :-| | 2to3| 1.227% | -0.084% | +4.452% | +7.996% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method| 1.589% | -0.412% | +22.315% | +11.703% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method_slots| 0.968% | +0.206% | +23.769% | +11.695% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_method_unknown| 1.059% | +0.086% | +22.368% | +9.792% | +-----+------------------------+--------+------------+------------+------------+ | :-| | call_simple| 3.153% | +1.416% | +8.945% | +7.342% | +-----+------------------------+--------+------------+------------+------------+ | :-| | chameleon| 1.203% | +0.359% | +11.727% | +7.864% | +-----+------------------------+--------+------------+------------+------------+ | :-| | chaos| 1.865% | -0.398% | +7.445% | +9.556% | +-----+------------------------+--------+------------+------------+------------+ | :-| | crypto_pyaes| 0.494% | +0.352% | +4.160% | +4.163% | +-----+------------------------+--------+------------+------------+------------+ | :-| | deltablue| 3.356% | +0.289% | +7.313% | +18.108% | +-----+------------------------+--------+------------+------------+------------+ | :-| | django_template| 6.588% | -1.479% | +7.015% | +13.719% | +-----+------------------------+--------+------------+------------+------------+ | :-| | dulwich_log| 1.295% | -0.008% | +3.306% | +6.399% | +-----+------------------------+--------+------------+------------+------------+ | :-| | fannkuch| 0.781% | -0.237% | +3.511% | +6.551% | +-----+------------------------+--------+------------+------------+------------+ | :-| | float| 0.988% | -0.157% | +3.066% | +6.909% | +-----+------------------------+--------+------------+------------+------------+ | :-| | genshi_text| 0.987% | -0.172% | +13.190% | +6.515% | +-----+------------------------+--------+------------+------------+------------+ | :-| | genshi_xml| 1.805% | -0.839% | +7.619% | +10.246% | +-----+------------------------+--------+------------+------------+------------+ | :-| | go| 1.032% | -0.276% | +4.818% | +12.082% | +-----+------------------------+--------+------------+------------+------------+ | :-| | hexiom| 0.537% | +0.054% | +9.768% | +11.396% | +-----+------------------------+--------+------------+------------+------------+ | :-| | html5lib| 2.586% | +0.191% | +8.222% | +9.719% | +-----+------------------------+--------+------------+------------+------------+ | :-| | json_dumps| 1.428% | +0.210% | +3.234% | +8.763% | +-----+------------------------+--------+------------+------------+------------+ | :-| | json_loads| 2.263% | -0.128% | -0.418% | +11.571% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_format| 1.895% | -0.218% | +7.739% | +11.980% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_silent| 2.974% | -0.498% | +47.104% | +11.039% | +-----+------------------------+--------+------------+------------+------------+ | :-| | logging_simple| 1.580% | -0.483% | +8.277% | +12.850% | +-----+------------------------+--------+------------+------------+------------+ | :-| | mako| 0.841% | -0.247% | +17.061% | +12.632% | +-----+------------------------+--------+------------+------------+------------+ | :-| | mdp| 2.269% | +0.009% | +9.242% | +9.857% | +-----+------------------------+--------+------------+------------+------------+ | :-| | meteor_contest| 1.602% | +0.194% | +2.995% | +5.723% | +-----+------------------------+--------+------------+------------+------------+ | :-| | nbody| 0.503% | +0.065% | -0.157% | -1.205% | +-----+------------------------+--------+------------+------------+------------+ | :-| | nqueens| 0.604% | +0.384% | +6.924% | +3.991% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pathlib| 1.121% | -0.153% | +6.742% | +7.567% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle| 1.142% | +0.169% | -0.905% | +22.743% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_dict| 2.177% | -0.383% | +1.291% | +19.980% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_list| 0.811% | -0.249% | +4.574% | +15.714% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pickle_pure_python| 3.810% | +0.570% | +10.831% | +9.759% | +-----+------------------------+--------+------------+------------+------------+ | :-| | pidigits| 0.302% | -0.102% | +0.068% | +10.115% | +-----+------------------------+--------+------------+------------+------------+ | :-| | python_startup| 0.125% | -0.160% | +11.076% | +4.492% | +-----+------------------------+--------+------------+------------+------------+ | :-| | python_startup_no_site| 0.095% | -0.144% | +2.406% | +4.662% | +-----+------------------------+--------+------------+------------+------------+ | :-| | raytrace| 1.524% | -0.020% | +9.269% | +13.265% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_compile| 5.240% | +0.007% | -1.624% | +12.732% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_dna| 1.308% | -0.134% | -1.596% | +9.897% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_effbot| 2.649% | -0.311% | -2.730% | +0.694% | +-----+------------------------+--------+------------+------------+------------+ | :-| | regex_v8| 2.169% | -0.175% | +5.999% | +8.003% | +-----+------------------------+--------+------------+------------+------------+ | :-| | richards| 1.571% | -0.419% | +6.961% | +14.381% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_fft| 0.792% | -0.259% | +2.529% | +0.730% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_lu| 1.846% | -0.384% | +27.753% | +6.014% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_monte_carlo| 1.521% | -0.180% | +4.043% | +5.182% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_sor| 0.394% | +0.070% | +15.549% | +5.794% | +-----+------------------------+--------+------------+------------+------------+ | :-| | scimark_sparse_mat_mult| 0.595% | -0.063% | +4.087% | -2.624% | +-----+------------------------+--------+------------+------------+------------+ | :-| | spectral_norm| 0.526% | -0.034% | +6.907% | +0.807% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlalchemy_declarative| 1.060% | -0.380% | +6.231% | +6.569% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlalchemy_imperative| 2.937% | +0.246% | +6.210% | +4.582% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sqlite_synth| 4.478% | -1.028% | +17.712% | +11.983% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_expand| 2.611% | -0.097% | +10.737% | +8.232% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_integrate| 2.006% | -0.672% | +8.865% | +6.268% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_str| 4.266% | -0.688% | +10.141% | +8.873% | +-----+------------------------+--------+------------+------------+------------+ | :-| | sympy_sum| 5.409% | -0.541% | +10.015% | +8.952% | +-----+------------------------+--------+------------+------------+------------+ | :-| | telco| 5.567% | -0.447% | +22.784% | +9.853% | +-----+------------------------+--------+------------+------------+------------+ | :-| | tornado_http| 1.053% | -0.178% | +5.349% | +7.120% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpack_sequence| 1.116% | -0.221% | +2.151% | -0.695% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle| 5.144% | +4.937% | +10.764% | +17.504% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle_list| 0.459% | -0.100% | +4.975% | +11.774% | +-----+------------------------+--------+------------+------------+------------+ | :-| | unpickle_pure_python| 2.002% | +1.558% | +7.901% | +5.687% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_generate| 0.850% | -0.172% | +6.375% | +6.911% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_iterparse| 2.198% | -0.029% | +2.995% | +6.785% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_parse| 2.486% | +0.107% | -5.656% | +11.745% | +-----+------------------------+--------+------------+------------+------------+ | :-| | xml_etree_process| 1.192% | -0.072% | +7.661% | +7.593% | +-----+------------------------+--------+------------+------------+------------+ * Relative Standard Deviation (Standard Deviation/Average) If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/65-flat-results-for-python-master-bra… Our lab does a nightly source pull and build of the Python project and measures performance changes against the previous stable version and the previous nightly measurement. This is provided as a service to the community so that quality issues with current hardware can be identified quickly. Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.

1 0

bpo-18835: Cleanup pymalloc (#4200)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/9ed83c40855b57c10988f76770a4eb825e… commit: 9ed83c40855b57c10988f76770a4eb825e034cd8 branch: master author: Victor Stinner <victor.stinner(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T12:18:10-07:00 summary: bpo-18835: Cleanup pymalloc (#4200) Cleanup pymalloc: * Rename _PyObject_Alloc() to pymalloc_alloc() * Rename _PyObject_FreeImpl() to pymalloc_free() * Rename _PyObject_Realloc() to pymalloc_realloc() * pymalloc_alloc() and pymalloc_realloc() don't fallback on the raw allocator anymore, it now must be done by the caller * Add "success" and "failed" labels to pymalloc_alloc() and pymalloc_free() * pymalloc_alloc() and pymalloc_free() don't update num_allocated_blocks anymore: it should be done in the caller * _PyObject_Calloc() is now responsible to fill the memory block allocated by pymalloc with zeros * Simplify pymalloc_alloc() prototype * _PyObject_Realloc() now calls _PyObject_Malloc() rather than calling directly pymalloc_alloc() _PyMem_DebugRawAlloc() and _PyMem_DebugRawRealloc(): * document the layout of a memory block * don't increase the serial number if the allocation failed * check for integer overflow before computing the total size * add a 'data' variable to make the code easiler to follow test_setallocators() of _testcapimodule.c now test also the context. files: M Modules/_testcapimodule.c M Objects/obmalloc.c diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 899c3d93701..1f71a09974e 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -3273,34 +3273,39 @@ typedef struct { void *realloc_ptr; size_t realloc_new_size; void *free_ptr; + void *ctx; } alloc_hook_t; -static void* hook_malloc (void* ctx, size_t size) +static void* hook_malloc(void* ctx, size_t size) { alloc_hook_t *hook = (alloc_hook_t *)ctx; + hook->ctx = ctx; hook->malloc_size = size; return hook->alloc.malloc(hook->alloc.ctx, size); } -static void* hook_calloc (void* ctx, size_t nelem, size_t elsize) +static void* hook_calloc(void* ctx, size_t nelem, size_t elsize) { alloc_hook_t *hook = (alloc_hook_t *)ctx; + hook->ctx = ctx; hook->calloc_nelem = nelem; hook->calloc_elsize = elsize; return hook->alloc.calloc(hook->alloc.ctx, nelem, elsize); } -static void* hook_realloc (void* ctx, void* ptr, size_t new_size) +static void* hook_realloc(void* ctx, void* ptr, size_t new_size) { alloc_hook_t *hook = (alloc_hook_t *)ctx; + hook->ctx = ctx; hook->realloc_ptr = ptr; hook->realloc_new_size = new_size; return hook->alloc.realloc(hook->alloc.ctx, ptr, new_size); } -static void hook_free (void *ctx, void *ptr) +static void hook_free(void *ctx, void *ptr) { alloc_hook_t *hook = (alloc_hook_t *)ctx; + hook->ctx = ctx; hook->free_ptr = ptr; hook->alloc.free(hook->alloc.ctx, ptr); } @@ -3325,7 +3330,9 @@ test_setallocators(PyMemAllocatorDomain domain) PyMem_GetAllocator(domain, &hook.alloc); PyMem_SetAllocator(domain, &alloc); + /* malloc, realloc, free */ size = 42; + hook.ctx = NULL; switch(domain) { case PYMEM_DOMAIN_RAW: ptr = PyMem_RawMalloc(size); break; @@ -3334,11 +3341,18 @@ test_setallocators(PyMemAllocatorDomain domain) default: ptr = NULL; break; } +#define CHECK_CTX(FUNC) \ + if (hook.ctx != &hook) { \ + error_msg = FUNC " wrong context"; \ + goto fail; \ + } \ + hook.ctx = NULL; /* reset for next check */ + if (ptr == NULL) { error_msg = "malloc failed"; goto fail; } - + CHECK_CTX("malloc"); if (hook.malloc_size != size) { error_msg = "malloc invalid size"; goto fail; @@ -3357,7 +3371,7 @@ test_setallocators(PyMemAllocatorDomain domain) error_msg = "realloc failed"; goto fail; } - + CHECK_CTX("realloc"); if (hook.realloc_ptr != ptr || hook.realloc_new_size != size2) { error_msg = "realloc invalid parameters"; @@ -3371,11 +3385,13 @@ test_setallocators(PyMemAllocatorDomain domain) case PYMEM_DOMAIN_OBJ: PyObject_Free(ptr2); break; } + CHECK_CTX("free"); if (hook.free_ptr != ptr2) { error_msg = "free invalid pointer"; goto fail; } + /* calloc, free */ nelem = 2; elsize = 5; switch(domain) @@ -3390,12 +3406,13 @@ test_setallocators(PyMemAllocatorDomain domain) error_msg = "calloc failed"; goto fail; } - + CHECK_CTX("calloc"); if (hook.calloc_nelem != nelem || hook.calloc_elsize != elsize) { error_msg = "calloc invalid nelem or elsize"; goto fail; } + hook.free_ptr = NULL; switch(domain) { case PYMEM_DOMAIN_RAW: PyMem_RawFree(ptr); break; @@ -3403,6 +3420,12 @@ test_setallocators(PyMemAllocatorDomain domain) case PYMEM_DOMAIN_OBJ: PyObject_Free(ptr); break; } + CHECK_CTX("calloc free"); + if (hook.free_ptr != ptr) { + error_msg = "calloc free invalid pointer"; + goto fail; + } + Py_INCREF(Py_None); res = Py_None; goto finally; @@ -3413,6 +3436,8 @@ test_setallocators(PyMemAllocatorDomain domain) finally: PyMem_SetAllocator(domain, &hook.alloc); return res; + +#undef CHECK_CTX } static PyObject * diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 1485172102d..b92116cd554 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -18,7 +18,7 @@ extern void _PyMem_DumpTraceback(int fd, const void *ptr); static void* _PyMem_DebugRawMalloc(void *ctx, size_t size); static void* _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize); static void* _PyMem_DebugRawRealloc(void *ctx, void *ptr, size_t size); -static void _PyMem_DebugRawFree(void *ctx, void *p); +static void _PyMem_DebugRawFree(void *ctx, void *ptr); static void* _PyMem_DebugMalloc(void *ctx, size_t size); static void* _PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize); @@ -444,6 +444,7 @@ PyMem_RawFree(void *ptr) _PyMem_Raw.free(_PyMem_Raw.ctx, ptr); } + void * PyMem_Malloc(size_t size) { @@ -477,6 +478,7 @@ PyMem_Free(void *ptr) _PyMem.free(_PyMem.ctx, ptr); } + char * _PyMem_RawStrdup(const char *str) { @@ -556,6 +558,7 @@ PyObject_Free(void *ptr) static int running_on_valgrind = -1; #endif + Py_ssize_t _Py_GetAllocatedBlocks(void) { @@ -661,6 +664,7 @@ new_arena(void) return arenaobj; } + /* address_in_range(P, POOL) @@ -750,442 +754,289 @@ address_in_range(void *p, poolp pool) _PyRuntime.mem.arenas[arenaindex].address != 0; } + /*==========================================================================*/ -/* malloc. Note that nbytes==0 tries to return a non-NULL pointer, distinct - * from all other currently live pointers. This may not be possible. - */ +/* pymalloc allocator -/* - * The basic blocks are ordered by decreasing execution frequency, - * which minimizes the number of jumps in the most common cases, - * improves branching prediction and instruction scheduling (small - * block allocations typically result in a couple of instructions). - * Unless the optimizer reorders everything, being too smart... - */ + The basic blocks are ordered by decreasing execution frequency, + which minimizes the number of jumps in the most common cases, + improves branching prediction and instruction scheduling (small + block allocations typically result in a couple of instructions). + Unless the optimizer reorders everything, being too smart... -static void * -_PyObject_Alloc(int use_calloc, void *ctx, size_t nelem, size_t elsize) + Return 1 if pymalloc allocated memory and wrote the pointer into *ptr_p. + + Return 0 if pymalloc failed to allocate the memory block: on bigger + requests, on error in the code below (as a last chance to serve the request) + or when the max memory limit has been reached. */ +static int +pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes) { - size_t nbytes; pyblock *bp; poolp pool; poolp next; uint size; - _PyRuntime.mem.num_allocated_blocks++; - - assert(elsize == 0 || nelem <= PY_SSIZE_T_MAX / elsize); - nbytes = nelem * elsize; - #ifdef WITH_VALGRIND - if (UNLIKELY(running_on_valgrind == -1)) + if (UNLIKELY(running_on_valgrind == -1)) { running_on_valgrind = RUNNING_ON_VALGRIND; - if (UNLIKELY(running_on_valgrind)) - goto redirect; + } + if (UNLIKELY(running_on_valgrind)) { + return 0; + } #endif - if (nelem == 0 || elsize == 0) - goto redirect; + if (nbytes == 0) { + return 0; + } + if (nbytes > SMALL_REQUEST_THRESHOLD) { + return 0; + } - if ((nbytes - 1) < SMALL_REQUEST_THRESHOLD) { - LOCK(); + LOCK(); + /* + * Most frequent paths first + */ + size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT; + pool = _PyRuntime.mem.usedpools[size + size]; + if (pool != pool->nextpool) { /* - * Most frequent paths first + * There is a used pool for this size class. + * Pick up the head block of its free list. */ - size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT; - pool = _PyRuntime.mem.usedpools[size + size]; - if (pool != pool->nextpool) { - /* - * There is a used pool for this size class. - * Pick up the head block of its free list. - */ - ++pool->ref.count; - bp = pool->freeblock; - assert(bp != NULL); - if ((pool->freeblock = *(pyblock **)bp) != NULL) { - UNLOCK(); - if (use_calloc) - memset(bp, 0, nbytes); - return (void *)bp; - } - /* - * Reached the end of the free list, try to extend it. - */ - if (pool->nextoffset <= pool->maxnextoffset) { - /* There is room for another block. */ - pool->freeblock = (pyblock*)pool + - pool->nextoffset; - pool->nextoffset += INDEX2SIZE(size); - *(pyblock **)(pool->freeblock) = NULL; - UNLOCK(); - if (use_calloc) - memset(bp, 0, nbytes); - return (void *)bp; - } - /* Pool is full, unlink from used pools. */ - next = pool->nextpool; - pool = pool->prevpool; - next->prevpool = pool; - pool->nextpool = next; - UNLOCK(); - if (use_calloc) - memset(bp, 0, nbytes); - return (void *)bp; + ++pool->ref.count; + bp = pool->freeblock; + assert(bp != NULL); + if ((pool->freeblock = *(pyblock **)bp) != NULL) { + goto success; } - /* There isn't a pool of the right size class immediately - * available: use a free pool. + /* + * Reached the end of the free list, try to extend it. */ - if (_PyRuntime.mem.usable_arenas == NULL) { - /* No arena has a free pool: allocate a new arena. */ -#ifdef WITH_MEMORY_LIMITS - if (_PyRuntime.mem.narenas_currently_allocated >= MAX_ARENAS) { - UNLOCK(); - goto redirect; - } -#endif - _PyRuntime.mem.usable_arenas = new_arena(); - if (_PyRuntime.mem.usable_arenas == NULL) { - UNLOCK(); - goto redirect; - } - _PyRuntime.mem.usable_arenas->nextarena = - _PyRuntime.mem.usable_arenas->prevarena = NULL; - } - assert(_PyRuntime.mem.usable_arenas->address != 0); - - /* Try to get a cached free pool. */ - pool = _PyRuntime.mem.usable_arenas->freepools; - if (pool != NULL) { - /* Unlink from cached pools. */ - _PyRuntime.mem.usable_arenas->freepools = pool->nextpool; - - /* This arena already had the smallest nfreepools - * value, so decreasing nfreepools doesn't change - * that, and we don't need to rearrange the - * usable_arenas list. However, if the arena has - * become wholly allocated, we need to remove its - * arena_object from usable_arenas. - */ - --_PyRuntime.mem.usable_arenas->nfreepools; - if (_PyRuntime.mem.usable_arenas->nfreepools == 0) { - /* Wholly allocated: remove. */ - assert(_PyRuntime.mem.usable_arenas->freepools == NULL); - assert(_PyRuntime.mem.usable_arenas->nextarena == NULL || - _PyRuntime.mem.usable_arenas->nextarena->prevarena == - _PyRuntime.mem.usable_arenas); - - _PyRuntime.mem.usable_arenas = _PyRuntime.mem.usable_arenas->nextarena; - if (_PyRuntime.mem.usable_arenas != NULL) { - _PyRuntime.mem.usable_arenas->prevarena = NULL; - assert(_PyRuntime.mem.usable_arenas->address != 0); - } - } - else { - /* nfreepools > 0: it must be that freepools - * isn't NULL, or that we haven't yet carved - * off all the arena's pools for the first - * time. - */ - assert(_PyRuntime.mem.usable_arenas->freepools != NULL || - _PyRuntime.mem.usable_arenas->pool_address <= - (pyblock*)_PyRuntime.mem.usable_arenas->address + - ARENA_SIZE - POOL_SIZE); - } - init_pool: - /* Frontlink to used pools. */ - next = _PyRuntime.mem.usedpools[size + size]; /* == prev */ - pool->nextpool = next; - pool->prevpool = next; - next->nextpool = pool; - next->prevpool = pool; - pool->ref.count = 1; - if (pool->szidx == size) { - /* Luckily, this pool last contained blocks - * of the same size class, so its header - * and free list are already initialized. - */ - bp = pool->freeblock; - assert(bp != NULL); - pool->freeblock = *(pyblock **)bp; - UNLOCK(); - if (use_calloc) - memset(bp, 0, nbytes); - return (void *)bp; - } - /* - * Initialize the pool header, set up the free list to - * contain just the second block, and return the first - * block. - */ - pool->szidx = size; - size = INDEX2SIZE(size); - bp = (pyblock *)pool + POOL_OVERHEAD; - pool->nextoffset = POOL_OVERHEAD + (size << 1); - pool->maxnextoffset = POOL_SIZE - size; - pool->freeblock = bp + size; + if (pool->nextoffset <= pool->maxnextoffset) { + /* There is room for another block. */ + pool->freeblock = (pyblock*)pool + + pool->nextoffset; + pool->nextoffset += INDEX2SIZE(size); *(pyblock **)(pool->freeblock) = NULL; - UNLOCK(); - if (use_calloc) - memset(bp, 0, nbytes); - return (void *)bp; + goto success; } - /* Carve off a new pool. */ - assert(_PyRuntime.mem.usable_arenas->nfreepools > 0); - assert(_PyRuntime.mem.usable_arenas->freepools == NULL); - pool = (poolp)_PyRuntime.mem.usable_arenas->pool_address; - assert((pyblock*)pool <= (pyblock*)_PyRuntime.mem.usable_arenas->address + - ARENA_SIZE - POOL_SIZE); - pool->arenaindex = (uint)(_PyRuntime.mem.usable_arenas - _PyRuntime.mem.arenas); - assert(&_PyRuntime.mem.arenas[pool->arenaindex] == _PyRuntime.mem.usable_arenas); - pool->szidx = DUMMY_SIZE_IDX; - _PyRuntime.mem.usable_arenas->pool_address += POOL_SIZE; - --_PyRuntime.mem.usable_arenas->nfreepools; + /* Pool is full, unlink from used pools. */ + next = pool->nextpool; + pool = pool->prevpool; + next->prevpool = pool; + pool->nextpool = next; + goto success; + } + /* There isn't a pool of the right size class immediately + * available: use a free pool. + */ + if (_PyRuntime.mem.usable_arenas == NULL) { + /* No arena has a free pool: allocate a new arena. */ +#ifdef WITH_MEMORY_LIMITS + if (_PyRuntime.mem.narenas_currently_allocated >= MAX_ARENAS) { + goto failed; + } +#endif + _PyRuntime.mem.usable_arenas = new_arena(); + if (_PyRuntime.mem.usable_arenas == NULL) { + goto failed; + } + _PyRuntime.mem.usable_arenas->nextarena = + _PyRuntime.mem.usable_arenas->prevarena = NULL; + } + assert(_PyRuntime.mem.usable_arenas->address != 0); + + /* Try to get a cached free pool. */ + pool = _PyRuntime.mem.usable_arenas->freepools; + if (pool != NULL) { + /* Unlink from cached pools. */ + _PyRuntime.mem.usable_arenas->freepools = pool->nextpool; + + /* This arena already had the smallest nfreepools + * value, so decreasing nfreepools doesn't change + * that, and we don't need to rearrange the + * usable_arenas list. However, if the arena has + * become wholly allocated, we need to remove its + * arena_object from usable_arenas. + */ + --_PyRuntime.mem.usable_arenas->nfreepools; if (_PyRuntime.mem.usable_arenas->nfreepools == 0) { + /* Wholly allocated: remove. */ + assert(_PyRuntime.mem.usable_arenas->freepools == NULL); assert(_PyRuntime.mem.usable_arenas->nextarena == NULL || _PyRuntime.mem.usable_arenas->nextarena->prevarena == _PyRuntime.mem.usable_arenas); - /* Unlink the arena: it is completely allocated. */ + _PyRuntime.mem.usable_arenas = _PyRuntime.mem.usable_arenas->nextarena; if (_PyRuntime.mem.usable_arenas != NULL) { _PyRuntime.mem.usable_arenas->prevarena = NULL; assert(_PyRuntime.mem.usable_arenas->address != 0); } } + else { + /* nfreepools > 0: it must be that freepools + * isn't NULL, or that we haven't yet carved + * off all the arena's pools for the first + * time. + */ + assert(_PyRuntime.mem.usable_arenas->freepools != NULL || + _PyRuntime.mem.usable_arenas->pool_address <= + (pyblock*)_PyRuntime.mem.usable_arenas->address + + ARENA_SIZE - POOL_SIZE); + } - goto init_pool; + init_pool: + /* Frontlink to used pools. */ + next = _PyRuntime.mem.usedpools[size + size]; /* == prev */ + pool->nextpool = next; + pool->prevpool = next; + next->nextpool = pool; + next->prevpool = pool; + pool->ref.count = 1; + if (pool->szidx == size) { + /* Luckily, this pool last contained blocks + * of the same size class, so its header + * and free list are already initialized. + */ + bp = pool->freeblock; + assert(bp != NULL); + pool->freeblock = *(pyblock **)bp; + goto success; + } + /* + * Initialize the pool header, set up the free list to + * contain just the second block, and return the first + * block. + */ + pool->szidx = size; + size = INDEX2SIZE(size); + bp = (pyblock *)pool + POOL_OVERHEAD; + pool->nextoffset = POOL_OVERHEAD + (size << 1); + pool->maxnextoffset = POOL_SIZE - size; + pool->freeblock = bp + size; + *(pyblock **)(pool->freeblock) = NULL; + goto success; } - /* The small block allocator ends here. */ - -redirect: - /* Redirect the original request to the underlying (libc) allocator. - * We jump here on bigger requests, on error in the code above (as a - * last chance to serve the request) or when the max memory limit - * has been reached. - */ - { - void *result; - if (use_calloc) - result = PyMem_RawCalloc(nelem, elsize); - else - result = PyMem_RawMalloc(nbytes); - if (!result) - _PyRuntime.mem.num_allocated_blocks--; - return result; + /* Carve off a new pool. */ + assert(_PyRuntime.mem.usable_arenas->nfreepools > 0); + assert(_PyRuntime.mem.usable_arenas->freepools == NULL); + pool = (poolp)_PyRuntime.mem.usable_arenas->pool_address; + assert((pyblock*)pool <= (pyblock*)_PyRuntime.mem.usable_arenas->address + + ARENA_SIZE - POOL_SIZE); + pool->arenaindex = (uint)(_PyRuntime.mem.usable_arenas - _PyRuntime.mem.arenas); + assert(&_PyRuntime.mem.arenas[pool->arenaindex] == _PyRuntime.mem.usable_arenas); + pool->szidx = DUMMY_SIZE_IDX; + _PyRuntime.mem.usable_arenas->pool_address += POOL_SIZE; + --_PyRuntime.mem.usable_arenas->nfreepools; + + if (_PyRuntime.mem.usable_arenas->nfreepools == 0) { + assert(_PyRuntime.mem.usable_arenas->nextarena == NULL || + _PyRuntime.mem.usable_arenas->nextarena->prevarena == + _PyRuntime.mem.usable_arenas); + /* Unlink the arena: it is completely allocated. */ + _PyRuntime.mem.usable_arenas = _PyRuntime.mem.usable_arenas->nextarena; + if (_PyRuntime.mem.usable_arenas != NULL) { + _PyRuntime.mem.usable_arenas->prevarena = NULL; + assert(_PyRuntime.mem.usable_arenas->address != 0); + } } + + goto init_pool; + +success: + UNLOCK(); + assert(bp != NULL); + *ptr_p = (void *)bp; + return 1; + +failed: + UNLOCK(); + return 0; } + static void * _PyObject_Malloc(void *ctx, size_t nbytes) { - return _PyObject_Alloc(0, ctx, 1, nbytes); + void* ptr; + if (pymalloc_alloc(ctx, &ptr, nbytes)) { + _PyRuntime.mem.num_allocated_blocks++; + return ptr; + } + + ptr = PyMem_RawMalloc(nbytes); + if (ptr != NULL) { + _PyRuntime.mem.num_allocated_blocks++; + } + return ptr; } + static void * _PyObject_Calloc(void *ctx, size_t nelem, size_t elsize) { - return _PyObject_Alloc(1, ctx, nelem, elsize); + void* ptr; + + assert(elsize == 0 || nelem <= (size_t)PY_SSIZE_T_MAX / elsize); + size_t nbytes = nelem * elsize; + + if (pymalloc_alloc(ctx, &ptr, nbytes)) { + memset(ptr, 0, nbytes); + _PyRuntime.mem.num_allocated_blocks++; + return ptr; + } + + ptr = PyMem_RawCalloc(nelem, elsize); + if (ptr != NULL) { + _PyRuntime.mem.num_allocated_blocks++; + } + return ptr; } -/* free */ -static void -_PyObject_Free(void *ctx, void *p) +/* Free a memory block allocated by pymalloc_alloc(). + Return 1 if it was freed. + Return 0 if the block was not allocated by pymalloc_alloc(). */ +static int +pymalloc_free(void *ctx, void *p) { poolp pool; pyblock *lastfree; poolp next, prev; uint size; - if (p == NULL) /* free(NULL) has no effect */ - return; - - _PyRuntime.mem.num_allocated_blocks--; + assert(p != NULL); #ifdef WITH_VALGRIND - if (UNLIKELY(running_on_valgrind > 0)) - goto redirect; + if (UNLIKELY(running_on_valgrind > 0)) { + return 0; + } #endif pool = POOL_ADDR(p); - if (address_in_range(p, pool)) { - /* We allocated this address. */ - LOCK(); - /* Link p to the start of the pool's freeblock list. Since - * the pool had at least the p block outstanding, the pool - * wasn't empty (so it's already in a usedpools[] list, or - * was full and is in no list -- it's not in the freeblocks - * list in any case). - */ - assert(pool->ref.count > 0); /* else it was empty */ - *(pyblock **)p = lastfree = pool->freeblock; - pool->freeblock = (pyblock *)p; - if (lastfree) { - struct arena_object* ao; - uint nf; /* ao->nfreepools */ - - /* freeblock wasn't NULL, so the pool wasn't full, - * and the pool is in a usedpools[] list. - */ - if (--pool->ref.count != 0) { - /* pool isn't empty: leave it in usedpools */ - UNLOCK(); - return; - } - /* Pool is now empty: unlink from usedpools, and - * link to the front of freepools. This ensures that - * previously freed pools will be allocated later - * (being not referenced, they are perhaps paged out). - */ - next = pool->nextpool; - prev = pool->prevpool; - next->prevpool = prev; - prev->nextpool = next; - - /* Link the pool to freepools. This is a singly-linked - * list, and pool->prevpool isn't used there. - */ - ao = &_PyRuntime.mem.arenas[pool->arenaindex]; - pool->nextpool = ao->freepools; - ao->freepools = pool; - nf = ++ao->nfreepools; - - /* All the rest is arena management. We just freed - * a pool, and there are 4 cases for arena mgmt: - * 1. If all the pools are free, return the arena to - * the system free(). - * 2. If this is the only free pool in the arena, - * add the arena back to the `usable_arenas` list. - * 3. If the "next" arena has a smaller count of free - * pools, we have to "slide this arena right" to - * restore that usable_arenas is sorted in order of - * nfreepools. - * 4. Else there's nothing more to do. - */ - if (nf == ao->ntotalpools) { - /* Case 1. First unlink ao from usable_arenas. - */ - assert(ao->prevarena == NULL || - ao->prevarena->address != 0); - assert(ao ->nextarena == NULL || - ao->nextarena->address != 0); - - /* Fix the pointer in the prevarena, or the - * usable_arenas pointer. - */ - if (ao->prevarena == NULL) { - _PyRuntime.mem.usable_arenas = ao->nextarena; - assert(_PyRuntime.mem.usable_arenas == NULL || - _PyRuntime.mem.usable_arenas->address != 0); - } - else { - assert(ao->prevarena->nextarena == ao); - ao->prevarena->nextarena = - ao->nextarena; - } - /* Fix the pointer in the nextarena. */ - if (ao->nextarena != NULL) { - assert(ao->nextarena->prevarena == ao); - ao->nextarena->prevarena = - ao->prevarena; - } - /* Record that this arena_object slot is - * available to be reused. - */ - ao->nextarena = _PyRuntime.mem.unused_arena_objects; - _PyRuntime.mem.unused_arena_objects = ao; - - /* Free the entire arena. */ - _PyRuntime.obj.allocator_arenas.free(_PyRuntime.obj.allocator_arenas.ctx, - (void *)ao->address, ARENA_SIZE); - ao->address = 0; /* mark unassociated */ - --_PyRuntime.mem.narenas_currently_allocated; - - UNLOCK(); - return; - } - if (nf == 1) { - /* Case 2. Put ao at the head of - * usable_arenas. Note that because - * ao->nfreepools was 0 before, ao isn't - * currently on the usable_arenas list. - */ - ao->nextarena = _PyRuntime.mem.usable_arenas; - ao->prevarena = NULL; - if (_PyRuntime.mem.usable_arenas) - _PyRuntime.mem.usable_arenas->prevarena = ao; - _PyRuntime.mem.usable_arenas = ao; - assert(_PyRuntime.mem.usable_arenas->address != 0); - - UNLOCK(); - return; - } - /* If this arena is now out of order, we need to keep - * the list sorted. The list is kept sorted so that - * the "most full" arenas are used first, which allows - * the nearly empty arenas to be completely freed. In - * a few un-scientific tests, it seems like this - * approach allowed a lot more memory to be freed. - */ - if (ao->nextarena == NULL || - nf <= ao->nextarena->nfreepools) { - /* Case 4. Nothing to do. */ - UNLOCK(); - return; - } - /* Case 3: We have to move the arena towards the end - * of the list, because it has more free pools than - * the arena to its right. - * First unlink ao from usable_arenas. - */ - if (ao->prevarena != NULL) { - /* ao isn't at the head of the list */ - assert(ao->prevarena->nextarena == ao); - ao->prevarena->nextarena = ao->nextarena; - } - else { - /* ao is at the head of the list */ - assert(_PyRuntime.mem.usable_arenas == ao); - _PyRuntime.mem.usable_arenas = ao->nextarena; - } - ao->nextarena->prevarena = ao->prevarena; + if (!address_in_range(p, pool)) { + return 0; + } + /* We allocated this address. */ - /* Locate the new insertion point by iterating over - * the list, using our nextarena pointer. - */ - while (ao->nextarena != NULL && - nf > ao->nextarena->nfreepools) { - ao->prevarena = ao->nextarena; - ao->nextarena = ao->nextarena->nextarena; - } + LOCK(); - /* Insert ao at this point. */ - assert(ao->nextarena == NULL || - ao->prevarena == ao->nextarena->prevarena); - assert(ao->prevarena->nextarena == ao->nextarena); - - ao->prevarena->nextarena = ao; - if (ao->nextarena != NULL) - ao->nextarena->prevarena = ao; - - /* Verify that the swaps worked. */ - assert(ao->nextarena == NULL || - nf <= ao->nextarena->nfreepools); - assert(ao->prevarena == NULL || - nf > ao->prevarena->nfreepools); - assert(ao->nextarena == NULL || - ao->nextarena->prevarena == ao); - assert((_PyRuntime.mem.usable_arenas == ao && - ao->prevarena == NULL) || - ao->prevarena->nextarena == ao); - - UNLOCK(); - return; - } + /* Link p to the start of the pool's freeblock list. Since + * the pool had at least the p block outstanding, the pool + * wasn't empty (so it's already in a usedpools[] list, or + * was full and is in no list -- it's not in the freeblocks + * list in any case). + */ + assert(pool->ref.count > 0); /* else it was empty */ + *(pyblock **)p = lastfree = pool->freeblock; + pool->freeblock = (pyblock *)p; + if (!lastfree) { /* Pool was full, so doesn't currently live in any list: * link it to the front of the appropriate usedpools[] list. * This mimics LRU pool usage for new allocations and @@ -1197,93 +1048,274 @@ _PyObject_Free(void *ctx, void *p) size = pool->szidx; next = _PyRuntime.mem.usedpools[size + size]; prev = next->prevpool; + /* insert pool before next: prev <-> pool <-> next */ pool->nextpool = next; pool->prevpool = prev; next->prevpool = pool; prev->nextpool = pool; - UNLOCK(); + goto success; + } + + struct arena_object* ao; + uint nf; /* ao->nfreepools */ + + /* freeblock wasn't NULL, so the pool wasn't full, + * and the pool is in a usedpools[] list. + */ + if (--pool->ref.count != 0) { + /* pool isn't empty: leave it in usedpools */ + goto success; + } + /* Pool is now empty: unlink from usedpools, and + * link to the front of freepools. This ensures that + * previously freed pools will be allocated later + * (being not referenced, they are perhaps paged out). + */ + next = pool->nextpool; + prev = pool->prevpool; + next->prevpool = prev; + prev->nextpool = next; + + /* Link the pool to freepools. This is a singly-linked + * list, and pool->prevpool isn't used there. + */ + ao = &_PyRuntime.mem.arenas[pool->arenaindex]; + pool->nextpool = ao->freepools; + ao->freepools = pool; + nf = ++ao->nfreepools; + + /* All the rest is arena management. We just freed + * a pool, and there are 4 cases for arena mgmt: + * 1. If all the pools are free, return the arena to + * the system free(). + * 2. If this is the only free pool in the arena, + * add the arena back to the `usable_arenas` list. + * 3. If the "next" arena has a smaller count of free + * pools, we have to "slide this arena right" to + * restore that usable_arenas is sorted in order of + * nfreepools. + * 4. Else there's nothing more to do. + */ + if (nf == ao->ntotalpools) { + /* Case 1. First unlink ao from usable_arenas. + */ + assert(ao->prevarena == NULL || + ao->prevarena->address != 0); + assert(ao ->nextarena == NULL || + ao->nextarena->address != 0); + + /* Fix the pointer in the prevarena, or the + * usable_arenas pointer. + */ + if (ao->prevarena == NULL) { + _PyRuntime.mem.usable_arenas = ao->nextarena; + assert(_PyRuntime.mem.usable_arenas == NULL || + _PyRuntime.mem.usable_arenas->address != 0); + } + else { + assert(ao->prevarena->nextarena == ao); + ao->prevarena->nextarena = + ao->nextarena; + } + /* Fix the pointer in the nextarena. */ + if (ao->nextarena != NULL) { + assert(ao->nextarena->prevarena == ao); + ao->nextarena->prevarena = + ao->prevarena; + } + /* Record that this arena_object slot is + * available to be reused. + */ + ao->nextarena = _PyRuntime.mem.unused_arena_objects; + _PyRuntime.mem.unused_arena_objects = ao; + + /* Free the entire arena. */ + _PyRuntime.obj.allocator_arenas.free(_PyRuntime.obj.allocator_arenas.ctx, + (void *)ao->address, ARENA_SIZE); + ao->address = 0; /* mark unassociated */ + --_PyRuntime.mem.narenas_currently_allocated; + + goto success; + } + + if (nf == 1) { + /* Case 2. Put ao at the head of + * usable_arenas. Note that because + * ao->nfreepools was 0 before, ao isn't + * currently on the usable_arenas list. + */ + ao->nextarena = _PyRuntime.mem.usable_arenas; + ao->prevarena = NULL; + if (_PyRuntime.mem.usable_arenas) + _PyRuntime.mem.usable_arenas->prevarena = ao; + _PyRuntime.mem.usable_arenas = ao; + assert(_PyRuntime.mem.usable_arenas->address != 0); + + goto success; + } + + /* If this arena is now out of order, we need to keep + * the list sorted. The list is kept sorted so that + * the "most full" arenas are used first, which allows + * the nearly empty arenas to be completely freed. In + * a few un-scientific tests, it seems like this + * approach allowed a lot more memory to be freed. + */ + if (ao->nextarena == NULL || + nf <= ao->nextarena->nfreepools) { + /* Case 4. Nothing to do. */ + goto success; + } + /* Case 3: We have to move the arena towards the end + * of the list, because it has more free pools than + * the arena to its right. + * First unlink ao from usable_arenas. + */ + if (ao->prevarena != NULL) { + /* ao isn't at the head of the list */ + assert(ao->prevarena->nextarena == ao); + ao->prevarena->nextarena = ao->nextarena; + } + else { + /* ao is at the head of the list */ + assert(_PyRuntime.mem.usable_arenas == ao); + _PyRuntime.mem.usable_arenas = ao->nextarena; + } + ao->nextarena->prevarena = ao->prevarena; + + /* Locate the new insertion point by iterating over + * the list, using our nextarena pointer. + */ + while (ao->nextarena != NULL && nf > ao->nextarena->nfreepools) { + ao->prevarena = ao->nextarena; + ao->nextarena = ao->nextarena->nextarena; + } + + /* Insert ao at this point. */ + assert(ao->nextarena == NULL || ao->prevarena == ao->nextarena->prevarena); + assert(ao->prevarena->nextarena == ao->nextarena); + + ao->prevarena->nextarena = ao; + if (ao->nextarena != NULL) { + ao->nextarena->prevarena = ao; + } + + /* Verify that the swaps worked. */ + assert(ao->nextarena == NULL || nf <= ao->nextarena->nfreepools); + assert(ao->prevarena == NULL || nf > ao->prevarena->nfreepools); + assert(ao->nextarena == NULL || ao->nextarena->prevarena == ao); + assert((_PyRuntime.mem.usable_arenas == ao && ao->prevarena == NULL) + || ao->prevarena->nextarena == ao); + + goto success; + +success: + UNLOCK(); + return 1; +} + + +static void +_PyObject_Free(void *ctx, void *p) +{ + /* PyObject_Free(NULL) has no effect */ + if (p == NULL) { return; } -#ifdef WITH_VALGRIND -redirect: -#endif - /* We didn't allocate this address. */ - PyMem_RawFree(p); + _PyRuntime.mem.num_allocated_blocks--; + if (!pymalloc_free(ctx, p)) { + /* pymalloc didn't allocate this address */ + PyMem_RawFree(p); + } } -/* realloc. If p is NULL, this acts like malloc(nbytes). Else if nbytes==0, - * then as the Python docs promise, we do not treat this like free(p), and - * return a non-NULL result. - */ -static void * -_PyObject_Realloc(void *ctx, void *p, size_t nbytes) +/* pymalloc realloc. + + If nbytes==0, then as the Python docs promise, we do not treat this like + free(p), and return a non-NULL result. + + Return 1 if pymalloc reallocated memory and wrote the new pointer into + newptr_p. + + Return 0 if pymalloc didn't allocated p. */ +static int +pymalloc_realloc(void *ctx, void **newptr_p, void *p, size_t nbytes) { void *bp; poolp pool; size_t size; - if (p == NULL) - return _PyObject_Alloc(0, ctx, 1, nbytes); + assert(p != NULL); #ifdef WITH_VALGRIND /* Treat running_on_valgrind == -1 the same as 0 */ - if (UNLIKELY(running_on_valgrind > 0)) - goto redirect; + if (UNLIKELY(running_on_valgrind > 0)) { + return 0; + } #endif pool = POOL_ADDR(p); - if (address_in_range(p, pool)) { - /* We're in charge of this block */ - size = INDEX2SIZE(pool->szidx); - if (nbytes <= size) { - /* The block is staying the same or shrinking. If - * it's shrinking, there's a tradeoff: it costs - * cycles to copy the block to a smaller size class, - * but it wastes memory not to copy it. The - * compromise here is to copy on shrink only if at - * least 25% of size can be shaved off. - */ - if (4 * nbytes > 3 * size) { - /* It's the same, - * or shrinking and new/old > 3/4. - */ - return p; - } - size = nbytes; - } - bp = _PyObject_Alloc(0, ctx, 1, nbytes); - if (bp != NULL) { - memcpy(bp, p, size); - _PyObject_Free(ctx, p); + if (!address_in_range(p, pool)) { + /* pymalloc is not managing this block. + + If nbytes <= SMALL_REQUEST_THRESHOLD, it's tempting to try to take + over this block. However, if we do, we need to copy the valid data + from the C-managed block to one of our blocks, and there's no + portable way to know how much of the memory space starting at p is + valid. + + As bug 1185883 pointed out the hard way, it's possible that the + C-managed block is "at the end" of allocated VM space, so that a + memory fault can occur if we try to copy nbytes bytes starting at p. + Instead we punt: let C continue to manage this block. */ + return 0; + } + + /* pymalloc is in charge of this block */ + size = INDEX2SIZE(pool->szidx); + if (nbytes <= size) { + /* The block is staying the same or shrinking. + + If it's shrinking, there's a tradeoff: it costs cycles to copy the + block to a smaller size class, but it wastes memory not to copy it. + + The compromise here is to copy on shrink only if at least 25% of + size can be shaved off. */ + if (4 * nbytes > 3 * size) { + /* It's the same, or shrinking and new/old > 3/4. */ + *newptr_p = p; + return 1; } - return bp; + size = nbytes; } -#ifdef WITH_VALGRIND - redirect: -#endif - /* We're not managing this block. If nbytes <= - * SMALL_REQUEST_THRESHOLD, it's tempting to try to take over this - * block. However, if we do, we need to copy the valid data from - * the C-managed block to one of our blocks, and there's no portable - * way to know how much of the memory space starting at p is valid. - * As bug 1185883 pointed out the hard way, it's possible that the - * C-managed block is "at the end" of allocated VM space, so that - * a memory fault can occur if we try to copy nbytes bytes starting - * at p. Instead we punt: let C continue to manage this block. - */ - if (nbytes) - return PyMem_RawRealloc(p, nbytes); - /* C doesn't define the result of realloc(p, 0) (it may or may not - * return NULL then), but Python's docs promise that nbytes==0 never - * returns NULL. We don't pass 0 to realloc(), to avoid that endcase - * to begin with. Even then, we can't be sure that realloc() won't - * return NULL. - */ - bp = PyMem_RawRealloc(p, 1); - return bp ? bp : p; + + bp = _PyObject_Malloc(ctx, nbytes); + if (bp != NULL) { + memcpy(bp, p, size); + _PyObject_Free(ctx, p); + } + *newptr_p = bp; + return 1; +} + + +static void * +_PyObject_Realloc(void *ctx, void *ptr, size_t nbytes) +{ + void *ptr2; + + if (ptr == NULL) { + return _PyObject_Malloc(ctx, nbytes); + } + + if (pymalloc_realloc(ctx, &ptr2, ptr, nbytes)) { + return ptr2; + } + + return PyMem_RawRealloc(ptr, nbytes); } #else /* ! WITH_PYMALLOC */ @@ -1386,37 +1418,53 @@ static void * _PyMem_DebugRawAlloc(int use_calloc, void *ctx, size_t nbytes) { debug_alloc_api_t *api = (debug_alloc_api_t *)ctx; - uint8_t *p; /* base address of malloc'ed block */ - uint8_t *tail; /* p + 2*SST + nbytes == pointer to tail pad bytes */ - size_t total; /* nbytes + 4*SST */ + uint8_t *p; /* base address of malloc'epad d block */ + uint8_t *data; /* p + 2*SST == pointer to data bytes */ + uint8_t *tail; /* data + nbytes == pointer to tail pad bytes */ + size_t total; /* 2 * SST + nbytes + 2 * SST */ - bumpserialno(); - total = nbytes + 4*SST; - if (nbytes > PY_SSIZE_T_MAX - 4*SST) - /* overflow: can't represent total as a Py_ssize_t */ + if (nbytes > (size_t)PY_SSIZE_T_MAX - 4 * SST) { + /* integer overflow: can't represent total as a Py_ssize_t */ return NULL; + } + total = nbytes + 4 * SST; - if (use_calloc) + /* Layout: [SSSS IFFF CCCC...CCCC FFFF NNNN] + * ^--- p ^--- data ^--- tail + S: nbytes stored as size_t + I: API identifier (1 byte) + F: Forbidden bytes (size_t - 1 bytes before, size_t bytes after) + C: Clean bytes used later to store actual data + N: Serial number stored as size_t */ + + if (use_calloc) { p = (uint8_t *)api->alloc.calloc(api->alloc.ctx, 1, total); - else + } + else { p = (uint8_t *)api->alloc.malloc(api->alloc.ctx, total); - if (p == NULL) + } + if (p == NULL) { return NULL; + } + data = p + 2*SST; + + bumpserialno(); /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */ write_size_t(p, nbytes); p[SST] = (uint8_t)api->api_id; memset(p + SST + 1, FORBIDDENBYTE, SST-1); - if (nbytes > 0 && !use_calloc) - memset(p + 2*SST, CLEANBYTE, nbytes); + if (nbytes > 0 && !use_calloc) { + memset(data, CLEANBYTE, nbytes); + } /* at tail, write pad (SST bytes) and serialno (SST bytes) */ - tail = p + 2*SST + nbytes; + tail = data + nbytes; memset(tail, FORBIDDENBYTE, SST); write_size_t(tail + SST, _PyRuntime.mem.serialno); - return p + 2*SST; + return data; } static void * @@ -1429,11 +1477,12 @@ static void * _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize) { size_t nbytes; - assert(elsize == 0 || nelem <= PY_SSIZE_T_MAX / elsize); + assert(elsize == 0 || nelem <= (size_t)PY_SSIZE_T_MAX / elsize); nbytes = nelem * elsize; return _PyMem_DebugRawAlloc(1, ctx, nbytes); } + /* The debug free first checks the 2*SST bytes on each end for sanity (in particular, that the FORBIDDENBYTEs with the api ID are still intact). Then fills the original bytes with DEADBYTE. @@ -1442,63 +1491,73 @@ _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize) static void _PyMem_DebugRawFree(void *ctx, void *p) { + /* PyMem_Free(NULL) has no effect */ + if (p == NULL) { + return; + } + debug_alloc_api_t *api = (debug_alloc_api_t *)ctx; uint8_t *q = (uint8_t *)p - 2*SST; /* address returned from malloc */ size_t nbytes; - if (p == NULL) - return; _PyMem_DebugCheckAddress(api->api_id, p); nbytes = read_size_t(q); - nbytes += 4*SST; - if (nbytes > 0) - memset(q, DEADBYTE, nbytes); + nbytes += 4 * SST; + memset(q, DEADBYTE, nbytes); api->alloc.free(api->alloc.ctx, q); } + static void * _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes) { + if (p == NULL) { + return _PyMem_DebugRawAlloc(0, ctx, nbytes); + } + debug_alloc_api_t *api = (debug_alloc_api_t *)ctx; - uint8_t *q = (uint8_t *)p; - uint8_t *tail; - size_t total; /* nbytes + 4*SST */ + uint8_t *q; /* base address of malloc'epad d block */ + uint8_t *data; /* p + 2*SST == pointer to data bytes */ + uint8_t *tail; /* data + nbytes == pointer to tail pad bytes */ + size_t total; /* 2 * SST + nbytes + 2 * SST */ size_t original_nbytes; int i; - if (p == NULL) - return _PyMem_DebugRawAlloc(0, ctx, nbytes); - _PyMem_DebugCheckAddress(api->api_id, p); - bumpserialno(); + + q = (uint8_t *)p; original_nbytes = read_size_t(q - 2*SST); - total = nbytes + 4*SST; - if (nbytes > PY_SSIZE_T_MAX - 4*SST) - /* overflow: can't represent total as a Py_ssize_t */ + if (nbytes > (size_t)PY_SSIZE_T_MAX - 4*SST) { + /* integer overflow: can't represent total as a Py_ssize_t */ return NULL; + } + total = nbytes + 4*SST; /* Resize and add decorations. */ q = (uint8_t *)api->alloc.realloc(api->alloc.ctx, q - 2*SST, total); - if (q == NULL) + if (q == NULL) { return NULL; + } + bumpserialno(); write_size_t(q, nbytes); assert(q[SST] == (uint8_t)api->api_id); - for (i = 1; i < SST; ++i) + for (i = 1; i < SST; ++i) { assert(q[SST + i] == FORBIDDENBYTE); - q += 2*SST; + } + data = q + 2*SST; - tail = q + nbytes; + tail = data + nbytes; memset(tail, FORBIDDENBYTE, SST); write_size_t(tail + SST, _PyRuntime.mem.serialno); if (nbytes > original_nbytes) { /* growing: mark new extra memory clean */ - memset(q + original_nbytes, CLEANBYTE, + memset(data + original_nbytes, CLEANBYTE, nbytes - original_nbytes); } - return q; + return data; } static void @@ -1523,6 +1582,7 @@ _PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize) return _PyMem_DebugRawCalloc(ctx, nelem, elsize); } + static void _PyMem_DebugFree(void *ctx, void *ptr) { @@ -1530,6 +1590,7 @@ _PyMem_DebugFree(void *ctx, void *ptr) _PyMem_DebugRawFree(ctx, ptr); } + static void * _PyMem_DebugRealloc(void *ctx, void *ptr, size_t nbytes) {

1 0

bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4202)
by Serhiy Storchaka 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/e7531e54bf195b8d3ed35b4138901c82f7… commit: e7531e54bf195b8d3ed35b4138901c82f7ed794c branch: 2.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2017-10-31T20:16:11+02:00 summary: bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4202) * Fix a compilation error on FreeBSD. * Fix the data attribute size on Mac OS X. (cherry picked from commit 2298fad5ff907dd48ea0fb5c71fa22334ef28c6b) files: M Modules/selectmodule.c diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index 12601713886..6cd32cd8a93 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1285,12 +1285,12 @@ static PyTypeObject kqueue_queue_Type; # define FFLAGS_FMT_UNIT "I" #endif -#ifdef __FreeBSD__ -# define DATA_TYPE T_INTPTRT -# define DATA_FMT_UNIT INTPTR_FMT_UNIT -#else +#if defined(__NetBSD__) || defined(__OpenBSD__) # define DATA_TYPE T_INT64 # define DATA_FMT_UNIT INT64_FMT_UNIT +#else +# define DATA_TYPE T_INTPTRT +# define DATA_FMT_UNIT INTPTRT_FMT_UNIT #endif /* Unfortunately, we can't store python objects in udata, because

1 0

bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4201)
by Serhiy Storchaka 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/84e252b79eed94bc9e9175f82191322c89… commit: 84e252b79eed94bc9e9175f82191322c89e489ad branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2017-10-31T20:16:07+02:00 summary: bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (GH-4196) (#4201) * Fix a compilation error on FreeBSD. * Fix the data attribute size on Mac OS X. (cherry picked from commit 2298fad5ff907dd48ea0fb5c71fa22334ef28c6b) files: M Modules/selectmodule.c diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index 3b9c4d65c1d..7e81f1904c0 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1841,12 +1841,12 @@ static PyTypeObject kqueue_queue_Type; # define FFLAGS_FMT_UNIT "I" #endif -#ifdef __FreeBSD__ -# define DATA_TYPE T_INTPTRT -# define DATA_FMT_UNIT INTPTR_FMT_UNIT -#else +#if defined(__NetBSD__) || defined(__OpenBSD__) # define DATA_TYPE T_INT64 # define DATA_FMT_UNIT INT64_FMT_UNIT +#else +# define DATA_TYPE T_INTPTRT +# define DATA_FMT_UNIT INTPTRT_FMT_UNIT #endif /* Unfortunately, we can't store python objects in udata, because

1 0

bpo-20064: Document PyObject_Malloc() (#4204)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/52ba7b447f41dad2754ddbc50ed97413b5… commit: 52ba7b447f41dad2754ddbc50ed97413b557bbe1 branch: 2.7 author: Victor Stinner <victor.stinner(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T10:08:28-07:00 summary: bpo-20064: Document PyObject_Malloc() (#4204) Document the following functions: * PyObject_Malloc() * PyObject_Realloc() * PyObject_Free() Document also the pymalloc allocator. files: M Doc/c-api/memory.rst M Objects/obmalloc.c diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 54655719045..258898d8702 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -155,6 +155,88 @@ versions and is therefore deprecated in extension modules. :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. +Object allocators +================= + +The following function sets, modeled after the ANSI C standard, but specifying +behavior when requesting zero bytes, are available for allocating and releasing +memory from the Python heap. + +By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. + +.. warning:: + + The :term:`GIL <global interpreter lock>` must be held when using these + functions. + +.. c:function:: void* PyObject_Malloc(size_t n) + + Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + allocated memory, or *NULL* if the request fails. + + Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as + if ``PyObject_Malloc(1)`` had been called instead. The memory will not have + been initialized in any way. + + +.. c:function:: void* PyObject_Realloc(void *p, size_t n) + + Resizes the memory block pointed to by *p* to *n* bytes. The contents will be + unchanged to the minimum of the old and the new sizes. + + If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n* + is equal to zero, the memory block is resized but is not freed, and the + returned pointer is non-*NULL*. + + Unless *p* is *NULL*, it must have been returned by a previous call to + :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`. + + If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains + a valid pointer to the previous memory area. + + +.. c:function:: void PyObject_Free(void *p) + + Frees the memory block pointed to by *p*, which must have been returned by a + previous call to :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or + :c:func:`PyObject_Calloc`. Otherwise, or if ``PyObject_Free(p)`` has been called + before, undefined behavior occurs. + + If *p* is *NULL*, no operation is performed. + + +In addition, the following macro sets are provided: + +* :c:func:`PyObject_MALLOC`: alias to :c:func:`PyObject_Malloc` +* :c:func:`PyObject_REALLOC`: alias to :c:func:`PyObject_Realloc` +* :c:func:`PyObject_FREE`: alias to :c:func:`PyObject_Free` +* :c:func:`PyObject_Del`: alias to :c:func:`PyObject_Free` +* :c:func:`PyObject_DEL`: alias to :c:func:`PyObject_FREE` (so finally an alias + to :c:func:`PyObject_Free`) + + +.. _pymalloc: + +The pymalloc allocator +====================== + +Python has a *pymalloc* allocator optimized for small objects (smaller or equal +to 512 bytes) with a short lifetime. It uses memory mappings called "arenas" +with a fixed size of 256 KiB. It falls back to :c:func:`malloc` and +:c:func:`realloc` for allocations larger than 512 bytes. + +*pymalloc* is the default allocator of :c:func:`PyObject_Malloc`. + +The arena allocator uses the following functions: + +* :c:func:`mmap` and :c:func:`munmap` if available, +* :c:func:`malloc` and :c:func:`free` otherwise. + +.. versionchanged:: 2.7.7 + The threshold changed from 256 to 512 bytes. The arena allocator now + uses :c:func:`mmap` if available. + + .. _memoryexamples: Examples diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 1bb1866dc6f..5655554b82d 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -214,7 +214,7 @@ static int running_on_valgrind = -1; * Arenas are allocated with mmap() on systems supporting anonymous memory * mappings to reduce heap fragmentation. */ -#define ARENA_SIZE (256 << 10) /* 256KB */ +#define ARENA_SIZE (256 << 10) /* 256KiB */ #ifdef WITH_MEMORY_LIMITS #define MAX_ARENAS (SMALL_MEMORY_LIMIT / ARENA_SIZE)

1 0

bpo-20064: Document PyObject_Malloc() (GH-4199) (#4203)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/8543ce8ffd57d770b57fe653e0ab7fada1… commit: 8543ce8ffd57d770b57fe653e0ab7fada1a4c343 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Victor Stinner <victor.stinner(a)gmail.com> date: 2017-10-31T09:56:11-07:00 summary: bpo-20064: Document PyObject_Malloc() (GH-4199) (#4203) Document the following functions: * PyObject_Malloc() * PyObject_Calloc() * PyObject_Realloc() * PyObject_Free() Fix also PyMem_RawFree() documentation. (cherry picked from commit ec2cbdd1dff2c51788136480b2085e77506ebf34) files: M Doc/c-api/memory.rst diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 873fb2ac1d3..73bec7c95a1 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -150,7 +150,7 @@ The default raw memory block allocator uses the following functions: Frees the memory block pointed to by *p*, which must have been returned by a previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or - :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_Free(p)`` has been + :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_RawFree(p)`` has been called before, undefined behavior occurs. If *p* is *NULL*, no operation is performed. @@ -263,6 +263,69 @@ versions and is therefore deprecated in extension modules. * ``PyMem_DEL(ptr)`` +Object allocators +================= + +The following function sets, modeled after the ANSI C standard, but specifying +behavior when requesting zero bytes, are available for allocating and releasing +memory from the Python heap. + +By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. + +.. warning:: + + The :term:`GIL <global interpreter lock>` must be held when using these + functions. + +.. c:function:: void* PyObject_Malloc(size_t n) + + Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + allocated memory, or *NULL* if the request fails. + + Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as + if ``PyObject_Malloc(1)`` had been called instead. The memory will not have + been initialized in any way. + + +.. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. + + Requesting zero elements or elements of size zero bytes returns a distinct + non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called + instead. + + .. versionadded:: 3.5 + + +.. c:function:: void* PyObject_Realloc(void *p, size_t n) + + Resizes the memory block pointed to by *p* to *n* bytes. The contents will be + unchanged to the minimum of the old and the new sizes. + + If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n* + is equal to zero, the memory block is resized but is not freed, and the + returned pointer is non-*NULL*. + + Unless *p* is *NULL*, it must have been returned by a previous call to + :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`. + + If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains + a valid pointer to the previous memory area. + + +.. c:function:: void PyObject_Free(void *p) + + Frees the memory block pointed to by *p*, which must have been returned by a + previous call to :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or + :c:func:`PyObject_Calloc`. Otherwise, or if ``PyObject_Free(p)`` has been called + before, undefined behavior occurs. + + If *p* is *NULL*, no operation is performed. + + Customize Memory Allocators ===========================

1 0

bpo-20064: Document PyObject_Malloc() (#4199)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/ec2cbdd1dff2c51788136480b2085e7750… commit: ec2cbdd1dff2c51788136480b2085e77506ebf34 branch: master author: Victor Stinner <victor.stinner(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T09:37:25-07:00 summary: bpo-20064: Document PyObject_Malloc() (#4199) Document the following functions: * PyObject_Malloc() * PyObject_Calloc() * PyObject_Realloc() * PyObject_Free() Fix also PyMem_RawFree() documentation. files: M Doc/c-api/memory.rst diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 22b9fe2a4a4..0a5b0ef44b4 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -150,7 +150,7 @@ The default raw memory block allocator uses the following functions: Frees the memory block pointed to by *p*, which must have been returned by a previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or - :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_Free(p)`` has been + :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_RawFree(p)`` has been called before, undefined behavior occurs. If *p* is *NULL*, no operation is performed. @@ -263,6 +263,69 @@ versions and is therefore deprecated in extension modules. * ``PyMem_DEL(ptr)`` +Object allocators +================= + +The following function sets, modeled after the ANSI C standard, but specifying +behavior when requesting zero bytes, are available for allocating and releasing +memory from the Python heap. + +By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. + +.. warning:: + + The :term:`GIL <global interpreter lock>` must be held when using these + functions. + +.. c:function:: void* PyObject_Malloc(size_t n) + + Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + allocated memory, or *NULL* if the request fails. + + Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as + if ``PyObject_Malloc(1)`` had been called instead. The memory will not have + been initialized in any way. + + +.. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. + + Requesting zero elements or elements of size zero bytes returns a distinct + non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called + instead. + + .. versionadded:: 3.5 + + +.. c:function:: void* PyObject_Realloc(void *p, size_t n) + + Resizes the memory block pointed to by *p* to *n* bytes. The contents will be + unchanged to the minimum of the old and the new sizes. + + If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n* + is equal to zero, the memory block is resized but is not freed, and the + returned pointer is non-*NULL*. + + Unless *p* is *NULL*, it must have been returned by a previous call to + :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`. + + If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains + a valid pointer to the previous memory area. + + +.. c:function:: void PyObject_Free(void *p) + + Frees the memory block pointed to by *p*, which must have been returned by a + previous call to :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or + :c:func:`PyObject_Calloc`. Otherwise, or if ``PyObject_Free(p)`` has been called + before, undefined behavior occurs. + + If *p* is *NULL*, no operation is performed. + + Customize Memory Allocators ===========================

1 0

bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (#4196)
by Serhiy Storchaka 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/2298fad5ff907dd48ea0fb5c71fa22334e… commit: 2298fad5ff907dd48ea0fb5c71fa22334ef28c6b branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T18:18:21+02:00 summary: bpo-31893: Fix errors in b9052a0f91d2e83bbc27267247a5920c82b242a3. (#4196) * Fix a compilation error on FreeBSD. * Fix the data attribute size on Mac OS X. files: M Modules/selectmodule.c diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index af265d89ff6..1cde6e849c4 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1840,12 +1840,12 @@ static PyTypeObject kqueue_queue_Type; # define FFLAGS_FMT_UNIT "I" #endif -#ifdef __FreeBSD__ -# define DATA_TYPE T_INTPTRT -# define DATA_FMT_UNIT INTPTR_FMT_UNIT -#else +#if defined(__NetBSD__) || defined(__OpenBSD__) # define DATA_TYPE T_INT64 # define DATA_FMT_UNIT INT64_FMT_UNIT +#else +# define DATA_TYPE T_INTPTRT +# define DATA_FMT_UNIT INTPTRT_FMT_UNIT #endif /* Unfortunately, we can't store python objects in udata, because

1 0

pythoninfo: add Py_DEBUG (#4198)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/afd055a59fe0291881fc2459215ce106e4… commit: afd055a59fe0291881fc2459215ce106e424da51 branch: master author: Victor Stinner <victor.stinner(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T08:41:10-07:00 summary: pythoninfo: add Py_DEBUG (#4198) files: M Lib/test/pythoninfo.py diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index 5447ab8274b..c238ef7b75c 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -115,6 +115,14 @@ def collect_sys(info_add): encoding = '%s/%s' % (encoding, errors) info_add('sys.%s.encoding' % name, encoding) + # Were we compiled --with-pydebug or with #define Py_DEBUG? + Py_DEBUG = hasattr(sys, 'gettotalrefcount') + if Py_DEBUG: + text = 'Yes (sys.gettotalrefcount() present)' + else: + text = 'No (sys.gettotalrefcount() missing)' + info_add('Py_DEBUG', text) + def collect_platform(info_add): import platform

1 0

test_capi.test__testcapi() becomes more verbose (#4197)
by Victor Stinner 31 Oct '17

31 Oct '17

https://github.com/python/cpython/commit/e1a470bd6ae87f9501b8830caa64e7a8df… commit: e1a470bd6ae87f9501b8830caa64e7a8df0a2302 branch: master author: Victor Stinner <victor.stinner(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2017-10-31T08:40:59-07:00 summary: test_capi.test__testcapi() becomes more verbose (#4197) Write the name of each subtest on a new line to help debugging when a test does crash Python. files: M Lib/test/test_capi.py diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 921735ef5d0..bb5b2a3b9f0 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -751,11 +751,16 @@ def callback(): class Test_testcapi(unittest.TestCase): def test__testcapi(self): + if support.verbose: + print() for name in dir(_testcapi): - if name.startswith('test_'): - with self.subTest("internal", name=name): - test = getattr(_testcapi, name) - test() + if not name.startswith('test_'): + continue + with self.subTest("internal", name=name): + if support.verbose: + print(f" {name}", flush=True) + test = getattr(_testcapi, name) + test() class PyMemDebugTests(unittest.TestCase):

1 0