Mailman 3 September 2011 - Python-checkins

Daily reference leaks (2b47f0146639): sum=0
by solipsis＠pitrou.net Sept. 27, 2011

Sept. 27, 2011

results for 2b47f0146639 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6GIDyV', '-x']

1 0

distutils2 (merge default -> python3): Merge devnotes update
by eric.araujo Sept. 26, 2011

Sept. 26, 2011

http://hg.python.org/distutils2/rev/51bb31c537cb changeset: 1196:51bb31c537cb branch: python3 parent: 1194:d9ce66d22bb5 parent: 1195:9e4d083c4ad6 user: Éric Araujo <merwok(a)netwok.org> date: Sat Sep 24 01:19:50 2011 +0200 summary: Merge devnotes update files: DEVNOTES.txt | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/DEVNOTES.txt b/DEVNOTES.txt --- a/DEVNOTES.txt +++ b/DEVNOTES.txt @@ -1,12 +1,16 @@ Notes … [View More]

1 0

peps: All open issues are now addressed.
by martin.v.loewis Sept. 26, 2011

Sept. 26, 2011

http://hg.python.org/peps/rev/4473ac0f72dc changeset: 3946:4473ac0f72dc user: Martin v. Löwis <martin(a)v.loewis.de> date: Mon Sep 26 12:32:32 2011 +0200 summary: All open issues are now addressed. files: pep-0393.txt | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-) diff --git a/pep-0393.txt b/pep-0393.txt --- a/pep-0393.txt +++ b/pep-0393.txt @@ -298,12 +298,6 @@ need to re-fetch the Py_UNICODE pointer after API calls; doing so will continue to work … [View More]

1 0

peps: Address GvR's comments.
by martin.v.loewis Sept. 26, 2011

Sept. 26, 2011

http://hg.python.org/peps/rev/81a533fd93f2 changeset: 3945:81a533fd93f2 user: Martin v. Löwis <martin(a)v.loewis.de> date: Mon Sep 26 12:25:49 2011 +0200 summary: Address GvR's comments. files: pep-0393.txt | 63 ++++++++++++++++++++++++++++++++++++++- 1 files changed, 61 insertions(+), 2 deletions(-) diff --git a/pep-0393.txt b/pep-0393.txt --- a/pep-0393.txt +++ b/pep-0393.txt @@ -19,7 +19,8 @@ representation in common cases, but give access to full UCS-4 on … [View More]all systems. For compatibility with existing APIs, several representations may exist in parallel; over time, this compatibility should be phased -out. +out. The distinction between narrow and wide Unicode builds is +dropped. An implementation of this PEP is available at [1]_. Rationale ========= @@ -206,10 +207,62 @@ converts a string to a char* (such as the ParseTuple functions) will use PyUnicode_AsUTF8 to compute a conversion. +New API +------- + +This section summarizes the API additions. + +Macros to access the internal representation of a Unicode object +(read-only): + +- PyUnicode_IS_COMPACT_ASCII(o), PyUnicode_IS_COMPACT(o), + PyUnicode_IS_READY(o) +- PyUnicode_GET_LENGTH(o) +- PyUnicode_KIND(o), PyUnicode_CHARACTER_SIZE(o), + PyUnicode_MAX_CHAR_VALUE(o) +- PyUnicode_DATA(o), PyUnicode_1BYTE_DATA(o), PyUnicode_2BYTE_DATA(o), + PyUnicode_4BYTE_DATA(o) + +Character access macros: + +- PyUnicode_READ(kind, data, index), PyUnicode_READ_CHAR(o, index) +- PyUnicode_WRITE(kind, data, index, value) + +Other macros: + +- PyUnicode_READY(o) +- PyUnicode_CONVERT_BYTES(from_type, tp_type, begin, end, to) + +String creation functions: + +- PyUnicode_New(size, maxchar) +- PyUnicode_FromKindAndData(kind, data, size) +- PyUnicode_Substring(o, start, end) +- PyUnicode_Chr(ch) + +Character access utility functions: + +- PyUnicode_CopyCharacters(to, to_start, from, from_start, how_many) +- PyUnicode_FindChar(str, ch, start, end, direction) + +Representation conversion: + +- PyUnicode_AsUCS4(o, buffer, buflen) +- PyUnicode_AsUCS4Copy(o) +- PyUnicode_AsUnicodeAndSize(o, size_out) +- PyUnicode_AsUTF8(o) +- PyUnicode_AsUTF8AndSize(o, size_out) + +UCS4 utility functions: + +- Py_UCS4_{strlen, strcpy, strcat, strncpy, strcmp, strncpy, strcmp, + strncmp, strchr, strrchr} + Stable ABI ---------- -None of the functions in this PEP become part of the stable ABI. +None of the functions in this PEP become part of the stable ABI +(PEP 384). GDB Debugging Hooks ------------------- @@ -363,6 +416,12 @@ PyUnicode_CopyCharacters help in analyzing and creating string objects, operating on indices instead of data pointers. +References +========== + +.. [1] PEP 393 branch + https://bitbucket.org/t0rsten/pep-393 + Copyright ========= -- Repository URL: http://hg.python.org/peps [View Less]

1 0

cpython: Issue #12981: rewrite multiprocessing_{sendfd,recvfd} in Python.
by charles-francois.natali Sept. 26, 2011

Sept. 26, 2011

http://hg.python.org/cpython/rev/95ee0df1e746 changeset: 72465:95ee0df1e746 user: Charles-François Natali <neologix(a)free.fr> date: Sat Sep 24 20:04:29 2011 +0200 summary: Issue #12981: rewrite multiprocessing_{sendfd,recvfd} in Python. files: Lib/multiprocessing/reduction.py | 21 +- Modules/_multiprocessing/multiprocessing.c | 128 +--------- Modules/_multiprocessing/multiprocessing.h | 10 - 3 files changed, 19 insertions(+), 140 deletions(-) … [View More]diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -39,6 +39,7 @@ import sys import socket import threading +import struct import _multiprocessing from multiprocessing import current_process @@ -51,7 +52,8 @@ # # -if not(sys.platform == 'win32' or hasattr(_multiprocessing, 'recvfd')): +if not(sys.platform == 'win32' or (hasattr(socket, 'CMSG_LEN') and + hasattr(socket, 'SCM_RIGHTS'))): raise ImportError('pickling of connections not supported') # @@ -77,10 +79,23 @@ else: def send_handle(conn, handle, destination_pid): - _multiprocessing.sendfd(conn.fileno(), handle) + with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s: + s.sendmsg([b'x'], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, + struct.pack("@i", handle))]) def recv_handle(conn): - return _multiprocessing.recvfd(conn.fileno()) + size = struct.calcsize("@i") + with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s: + msg, ancdata, flags, addr = s.recvmsg(1, socket.CMSG_LEN(size)) + try: + cmsg_level, cmsg_type, cmsg_data = ancdata[0] + if (cmsg_level == socket.SOL_SOCKET and + cmsg_type == socket.SCM_RIGHTS): + return struct.unpack("@i", cmsg_data[:size])[0] + except (ValueError, IndexError, struct.error): + pass + raise RuntimeError('Invalid data received') + # # Support for a per-process server thread which caches pickled handles diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c --- a/Modules/_multiprocessing/multiprocessing.c +++ b/Modules/_multiprocessing/multiprocessing.c @@ -8,11 +8,6 @@ #include "multiprocessing.h" -#ifdef SCM_RIGHTS - #define HAVE_FD_TRANSFER 1 -#else - #define HAVE_FD_TRANSFER 0 -#endif PyObject *create_win32_namespace(void); @@ -75,115 +70,7 @@ return FALSE; } -/* - * Unix only - */ - -#else /* !MS_WINDOWS */ - -#if HAVE_FD_TRANSFER - -/* Functions for transferring file descriptors between processes. - Reimplements some of the functionality of the fdcred - module at http://www.mca-ltd.com/resources/fdcred_1.tgz. */ -/* Based in http://resin.csoft.net/cgi-bin/man.cgi?section=3&topic=CMSG_DATA */ - -static PyObject * -multiprocessing_sendfd(PyObject *self, PyObject *args) -{ - int conn, fd, res; - struct iovec dummy_iov; - char dummy_char; - struct msghdr msg; - struct cmsghdr *cmsg; - union { - struct cmsghdr hdr; - unsigned char buf[CMSG_SPACE(sizeof(int))]; - } cmsgbuf; - - if (!PyArg_ParseTuple(args, "ii", &conn, &fd)) - return NULL; - - dummy_iov.iov_base = &dummy_char; - dummy_iov.iov_len = 1; - - memset(&msg, 0, sizeof(msg)); - msg.msg_control = &cmsgbuf.buf; - msg.msg_controllen = sizeof(cmsgbuf.buf); - msg.msg_iov = &dummy_iov; - msg.msg_iovlen = 1; - - cmsg = CMSG_FIRSTHDR(&msg); - cmsg->cmsg_len = CMSG_LEN(sizeof(int)); - cmsg->cmsg_level = SOL_SOCKET; - cmsg->cmsg_type = SCM_RIGHTS; - * (int *) CMSG_DATA(cmsg) = fd; - - Py_BEGIN_ALLOW_THREADS - res = sendmsg(conn, &msg, 0); - Py_END_ALLOW_THREADS - - if (res < 0) - return PyErr_SetFromErrno(PyExc_OSError); - Py_RETURN_NONE; -} - -static PyObject * -multiprocessing_recvfd(PyObject *self, PyObject *args) -{ - int conn, fd, res; - char dummy_char; - struct iovec dummy_iov; - struct msghdr msg = {0}; - struct cmsghdr *cmsg; - union { - struct cmsghdr hdr; - unsigned char buf[CMSG_SPACE(sizeof(int))]; - } cmsgbuf; - - if (!PyArg_ParseTuple(args, "i", &conn)) - return NULL; - - dummy_iov.iov_base = &dummy_char; - dummy_iov.iov_len = 1; - - memset(&msg, 0, sizeof(msg)); - msg.msg_control = &cmsgbuf.buf; - msg.msg_controllen = sizeof(cmsgbuf.buf); - msg.msg_iov = &dummy_iov; - msg.msg_iovlen = 1; - - cmsg = CMSG_FIRSTHDR(&msg); - cmsg->cmsg_level = SOL_SOCKET; - cmsg->cmsg_type = SCM_RIGHTS; - cmsg->cmsg_len = CMSG_LEN(sizeof(int)); - msg.msg_controllen = cmsg->cmsg_len; - - Py_BEGIN_ALLOW_THREADS - res = recvmsg(conn, &msg, 0); - Py_END_ALLOW_THREADS - - if (res < 0) - return PyErr_SetFromErrno(PyExc_OSError); - - if (msg.msg_controllen < CMSG_LEN(sizeof(int)) || - (cmsg = CMSG_FIRSTHDR(&msg)) == NULL || - cmsg->cmsg_level != SOL_SOCKET || - cmsg->cmsg_type != SCM_RIGHTS || - cmsg->cmsg_len < CMSG_LEN(sizeof(int))) { - /* If at least one control message is present, there should be - no room for any further data in the buffer. */ - PyErr_SetString(PyExc_RuntimeError, "No file descriptor received"); - return NULL; - } - - fd = * (int *) CMSG_DATA(cmsg); - return Py_BuildValue("i", fd); -} - -#endif /* HAVE_FD_TRANSFER */ - -#endif /* !MS_WINDOWS */ +#endif /* MS_WINDOWS */ /* @@ -212,16 +99,6 @@ {"address_of_buffer", multiprocessing_address_of_buffer, METH_O, "address_of_buffer(obj) -> int\n" "Return address of obj assuming obj supports buffer inteface"}, -#if HAVE_FD_TRANSFER - {"sendfd", multiprocessing_sendfd, METH_VARARGS, - "sendfd(sockfd, fd) -> None\n" - "Send file descriptor given by fd over the unix domain socket\n" - "whose file decriptor is sockfd"}, - {"recvfd", multiprocessing_recvfd, METH_VARARGS, - "recvfd(sockfd) -> fd\n" - "Receive a file descriptor over a unix domain socket\n" - "whose file decriptor is sockfd"}, -#endif {NULL} }; @@ -319,9 +196,6 @@ #ifdef HAVE_SEM_TIMEDWAIT ADD_FLAG(HAVE_SEM_TIMEDWAIT); #endif -#ifdef HAVE_FD_TRANSFER - ADD_FLAG(HAVE_FD_TRANSFER); -#endif #ifdef HAVE_BROKEN_SEM_GETVALUE ADD_FLAG(HAVE_BROKEN_SEM_GETVALUE); #endif diff --git a/Modules/_multiprocessing/multiprocessing.h b/Modules/_multiprocessing/multiprocessing.h --- a/Modules/_multiprocessing/multiprocessing.h +++ b/Modules/_multiprocessing/multiprocessing.h @@ -3,12 +3,6 @@ #define PY_SSIZE_T_CLEAN -#ifdef __sun -/* The control message API is only available on Solaris - if XPG 4.2 or later is requested. */ -#define _XOPEN_SOURCE 500 -#endif - #include "Python.h" #include "structmember.h" #include "pythread.h" @@ -29,10 +23,6 @@ # define SEM_VALUE_MAX LONG_MAX #else # include <fcntl.h> /* O_CREAT and O_EXCL */ -# include <netinet/in.h> -# include <sys/socket.h> -# include <sys/uio.h> -# include <arpa/inet.h> /* htonl() and ntohl() */ # if defined(HAVE_SEM_OPEN) && !defined(POSIX_SEMAPHORES_NOT_ENABLED) # include <semaphore.h> typedef sem_t *SEM_HANDLE; -- Repository URL: http://hg.python.org/cpython [View Less]

2 1

Daily reference leaks (2b47f0146639): sum=0
by solipsis＠pitrou.net Sept. 26, 2011

Sept. 26, 2011

results for 2b47f0146639 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogK8G21W', '-x']

1 0

peps: Update to current object layout.
by martin.v.loewis Sept. 25, 2011

Sept. 25, 2011

http://hg.python.org/peps/rev/a97dfa0fa127 changeset: 3944:a97dfa0fa127 user: Martin v. Löwis <martin(a)v.loewis.de> date: Sun Sep 25 22:58:13 2011 +0200 summary: Update to current object layout. files: pep-0393.txt | 191 ++++++++++++++++++++++---------------- 1 files changed, 112 insertions(+), 79 deletions(-) diff --git a/pep-0393.txt b/pep-0393.txt --- a/pep-0393.txt +++ b/pep-0393.txt @@ -47,52 +47,88 @@ For many strings (e.g. ASCII), multiple … [View More]representations may actually share memory (e.g. the shortest form may be shared with the UTF-8 form if all characters are ASCII). With such sharing, the overhead of -compatibility representations is reduced. +compatibility representations is reduced. If representations do share +data, it is also possible to omit structure fields, reducing the base +size of string objects. Specification ============= -The Unicode object structure is changed to this definition:: +Unicode structures are now defined as a hierarchy of structures, +namely:: typedef struct { PyObject_HEAD Py_ssize_t length; + Py_hash_t hash; + struct { + unsigned int interned:2; + unsigned int kind:2; + unsigned int compact:1; + unsigned int ascii:1; + unsigned int ready:1; + } state; + wchar_t *wstr; + } PyASCIIObject; + + typedef struct { + PyASCIIObject _base; + Py_ssize_t utf8_length; + char *utf8; + Py_ssize_t wstr_length; + } PyCompactUnicodeObject; + + typedef struct { + PyCompactUnicodeObject _base; union { void *any; Py_UCS1 *latin1; Py_UCS2 *ucs2; Py_UCS4 *ucs4; } data; - Py_hash_t hash; - int state; - Py_ssize_t utf8_length; - void *utf8; - Py_ssize_t wstr_length; - void *wstr; } PyUnicodeObject; -These fields have the following interpretations: +Objects for which both size and maximum character are known at +creation time are called "compact" unicode objects; character data +immediately follow the base structure. If the maximum character is +less than 128, they use the PyASCIIObject structure, and the UTF-8 +data, the UTF-8 length and the wstr length are the same as the length +and the ASCII data. For non-ASCII strings, the PyCompactObject +structure is used. Resizing compact objects is not supported. + +Objects for which the maximum character is not given at creation time +are called "legacy" objects, created through +PyUnicode_FromStringAndSize(NULL, length). They use the +PyUnicodeObject structure. Initially, their data is only in the wstr +pointer; when PyUnicode_READY is called, the data pointer (union) is +allocated. Resizing is possible as long PyUnicode_READY has not been +called. + +The fields have the following interpretations: - length: number of code points in the string (result of sq_length) -- data: shortest-form representation of the unicode string. - The string is null-terminated (in its respective representation). -- hash: same as in Python 3.2 -- state: - - * lowest 2 bits (mask 0x03) - interned-state (SSTATE_*) as in 3.2 - * next 2 bits (mask 0x0C) - form of str: - +- interned: interned-state (SSTATE_*) as in 3.2 +- kind: form of string + 00 => str is not initialized (data are in wstr) + 01 => 1 byte (Latin-1) + 10 => 2 byte (UCS-2) + 11 => 4 byte (UCS-4); - - * next bit (mask 0x10): 1 if str memory follows PyUnicodeObject - -- utf8_length, utf8: UTF-8 representation (null-terminated) +- compact: the object uses one of the compact representations + (implies ready) +- ascii: the object uses the PyASCIIObject representation + (implies compact and ready) +- ready: the canonical represenation is ready to be accessed through + PyUnicode_DATA and PyUnicode_GET_LENGTH. This is set either if the + object is compact, or the data pointer and length have been + initialized. - wstr_length, wstr: representation in platform's wchar_t (null-terminated). If wchar_t is 16-bit, this form may use surrogate pairs (in which cast wstr_length differs form length). + wstr_length differs from length only if there are surrogate pairs + in the representation. +- utf8_length, utf8: UTF-8 representation (null-terminated). +- data: shortest-form representation of the unicode string. + The string is null-terminated (in its respective representation). All three representations are optional, although the data form is considered the canonical representation which can be absent only @@ -111,10 +147,6 @@ BMP-not-Latin-1 characters if sizeof(wchar_t) is 2, and uses some non-BMP characters if sizeof(wchar_t) is 4). -If the string is created directly with the canonical representation -(see below), this representation doesn't take a separate memory block, -but is allocated right after the PyUnicodeObject struct. - String Creation --------------- @@ -140,12 +172,11 @@ or implicitly). Resizing a Unicode string remains possible until it is finalized. -PyUnicode_Ready() converts a string containing only a wstr +PyUnicode_READY() converts a string containing only a wstr representation into the canonical representation. Unless wstr and data can share the memory, the wstr representation is discarded after the -conversion. PyUnicode_FAST_READY() is a wrapper that avoids the -function call if the string is already ready. Both APIs return 0 -on success and -1 on failure. +conversion. The macro returns 0 on success and -1 on failure, which +happens in particular if the memory allocation fails. String Access ------------- @@ -175,9 +206,6 @@ converts a string to a char* (such as the ParseTuple functions) will use PyUnicode_AsUTF8 to compute a conversion. -PyUnicode_AsUnicode is deprecated; it computes the wstr representation -on first use. - Stable ABI ---------- @@ -189,27 +217,37 @@ about the internals of CPython's data types, include PyUnicodeObject instances. It will need to be slightly updated to track the change. +Deprecations, Removals, and Incompatibilities +--------------------------------------------- + +While the Py_UNICODE representation and APIs are deprecated with this +PEP, no removal of the respective APIs is scheduled. The APIs should +remain available at least five years after the PEP is accepted; before +they are removed, existing extension modules should be studied to find +out whether a sufficient majority of the open-source code on PyPI has +been ported to the new API. A reasonable motivation for using the +deprecated API even in new code is for code that shall work both on +Python 2 and Python 3. + +_PyUnicode_AsDefaultEncodedString is removed. It previously returned a +borrowed reference to an UTF-8-encoded bytes object. Since the unicode +object cannot anymore cache such a reference, implementing it without +leaking memory is not possible. No deprecation phase is provided, +since it was an API for internal use only. + +Extension modules using the legacy API may inadvertently call +PyUnicode_READY, by calling some API that requires that the object is +ready, and then continue accessing the (now invalid) Py_UNICODE +pointer. Such code will break with this PEP. The code was already +flawed in 3.2, as there is was no explicit guarantee that the +PyUnicode_AS_UNICODE result would stay valid after an API call (due to +the possiblity of string resizing). Modules that face this issue +need to re-fetch the Py_UNICODE pointer after API calls; doing +so will continue to work correctly in earlier Python versions. + Open Issues =========== -- When an application uses the legacy API, it may hold onto - the Py_UNICODE* representation, and yet start calling Unicode - APIs, which would call PyUnicode_Ready, invalidating the - Py_UNICODE* representation; this would be an incompatible change. - The following solutions can be considered: - - * accept it as an incompatible change. Applications using the - legacy API will have to fill out the Py_UNICODE buffer completely - before calling any API on the string under construction. - * require explicit PyUnicode_Ready calls in such applications; - fail with a fatal error if a non-ready string is ever read. - This would also be an incompatible change, but one that is - more easily detected during testing. - * as a compromise between these approaches, implicit PyUnicode_Ready - calls (i.e. those not deliberately following the construction of - a PyUnicode object) could produce a warning if they convert an - object. - - Which of the APIs created during the development of the PEP should be public? @@ -226,11 +264,6 @@ applications that care about this problem can be rewritten to use the data representation. -The question was raised whether the wchar_t representation is -discouraged, or scheduled for removal. This is not the intent of this -PEP; applications that use them will see a performance penalty, -though. Future versions of Python may consider to remove them. - Performance ----------- @@ -240,31 +273,31 @@ a reduction in memory usage. For small strings, the effects depend on the pointer size of the system, and the size of the Py_UNICODE/wchar_t type. The following table demonstrates this for various small ASCII -string sizes and platforms. +and Latin-1 string sizes and platforms. -+-------+---------------------------------+----------------+ -|string | Python 3.2 | This PEP | -|size +----------------+----------------+ | -| | 16-bit wchar_t | 32-bit wchar_t | | -| +---------+------+--------+-------+--------+-------+ -| | 32-bit |64-bit| 32-bit |64-bit | 32-bit |64-bit | -+-------+---------+------+--------+-------+--------+-------+ -|1 | 40 | 64 | 40 | 64 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|2 | 40 | 64 | 48 | 72 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|3 | 40 | 64 | 48 | 72 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|4 | 48 | 72 | 56 | 80 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|5 | 48 | 72 | 56 | 80 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|6 | 48 | 72 | 64 | 88 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|7 | 48 | 72 | 64 | 88 | 48 | 88 | -+-------+---------+------+--------+-------+--------+-------+ -|8 | 56 | 80 | 72 | 96 | 56 | 88 | -+-------+---------+------+--------+-------+--------+-------+ ++-------+---------------------------------+---------------------------------+ +|string | Python 3.2 | This PEP | +|size +----------------+----------------+----------------+----------------+ +| | 16-bit wchar_t | 32-bit wchar_t | ASCII | Latin-1 | +| +---------+------+--------+-------+--------+-------+--------+-------+ +| | 32-bit |64-bit| 32-bit |64-bit | 32-bit |64-bit | 32-bit |64-bit | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|1 | 32 | 64 | 40 | 64 | 32 | 56 | 40 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|2 | 40 | 64 | 40 | 72 | 32 | 56 | 40 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|3 | 40 | 64 | 48 | 72 | 32 | 56 | 40 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|4 | 40 | 72 | 48 | 80 | 32 | 56 | 48 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|5 | 40 | 72 | 56 | 80 | 32 | 56 | 48 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|6 | 48 | 72 | 56 | 88 | 32 | 56 | 48 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|7 | 48 | 72 | 64 | 88 | 32 | 56 | 48 | 80 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ +|8 | 48 | 80 | 64 | 96 | 40 | 64 | 48 | 88 | ++-------+---------+------+--------+-------+--------+-------+--------+-------+ The runtime effect is significantly affected by the API being used. After porting the relevant pieces of code to the new API, -- Repository URL: http://hg.python.org/peps [View Less]

1 0

cpython (3.2): Depend setobject.o and dictobject.o on stringlib/eq.h.
by martin.v.loewis Sept. 25, 2011

Sept. 25, 2011

http://hg.python.org/cpython/rev/4951ffc178db changeset: 72469:4951ffc178db branch: 3.2 parent: 72459:b378864d8ff3 user: Martin v. Löwis <martin(a)v.loewis.de> date: Sun Sep 25 17:36:11 2011 +0200 summary: Depend setobject.o and dictobject.o on stringlib/eq.h. files: Makefile.pre.in | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -641,6 +641,9 @@ … [View More]

1 0

cpython (merge 3.2 -> default): merged
by martin.v.loewis Sept. 25, 2011

Sept. 25, 2011

http://hg.python.org/cpython/rev/2b47f0146639 changeset: 72470:2b47f0146639 parent: 72468:3fb9464f9b02 parent: 72469:4951ffc178db user: Martin v. Löwis <martin(a)v.loewis.de> date: Sun Sep 25 17:36:31 2011 +0200 summary: merged files: Makefile.pre.in | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -641,6 +641,9 @@ Objects/unicodeobject.o: $(srcdir)/… [View More]

1 0

cpython: Issue #1621: Fix undefined behaviour from signed overflow in datetime module
by mark.dickinson Sept. 25, 2011

Sept. 25, 2011

http://hg.python.org/cpython/rev/3fb9464f9b02 changeset: 72468:3fb9464f9b02 user: Mark Dickinson <mdickinson(a)enthought.com> date: Sun Sep 25 15:34:32 2011 +0100 summary: Issue #1621: Fix undefined behaviour from signed overflow in datetime module hashes, array and list iterations, and get_integer (stringlib/string_format.h) files: Modules/_datetimemodule.c | 8 ++++---- Modules/arraymodule.c | 2 +- Objects/listobject.c | 6 ++… [View More]+--- Objects/stringlib/string_format.h | 14 ++++++-------- 4 files changed, 14 insertions(+), 16 deletions(-) diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -2746,13 +2746,13 @@ generic_hash(unsigned char *data, int len) { register unsigned char *p; - register Py_hash_t x; + register Py_uhash_t x; p = (unsigned char *) data; - x = *p << 7; + x = (Py_uhash_t)*p << 7; while (--len >= 0) - x = (1000003*x) ^ *p++; - x ^= len; + x = (1000003U*x) ^ (Py_uhash_t)*p++; + x ^= (Py_uhash_t)len; if (x == -1) x = -2; diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2351,7 +2351,7 @@ self->ob_item + (cur + 1) * itemsize, lim * itemsize); } - cur = start + slicelength * step; + cur = start + (size_t)slicelength * step; if (cur < (size_t)Py_SIZE(self)) { memmove(self->ob_item + (cur-slicelength) * itemsize, self->ob_item + cur * itemsize, diff --git a/Objects/listobject.c b/Objects/listobject.c --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2434,7 +2434,7 @@ src = self->ob_item; dest = ((PyListObject *)result)->ob_item; for (cur = start, i = 0; i < slicelength; - cur += step, i++) { + cur += (size_t)step, i++) { it = src[cur]; Py_INCREF(it); dest[i] = it; @@ -2525,7 +2525,7 @@ self->ob_item + cur + 1, lim * sizeof(PyObject *)); } - cur = start + slicelength*step; + cur = start + (size_t)slicelength * step; if (cur < (size_t)Py_SIZE(self)) { memmove(self->ob_item + cur - slicelength, self->ob_item + cur, @@ -2589,7 +2589,7 @@ selfitems = self->ob_item; seqitems = PySequence_Fast_ITEMS(seq); for (cur = start, i = 0; i < slicelength; - cur += step, i++) { + cur += (size_t)step, i++) { garbage[i] = selfitems[cur]; ins = seqitems[i]; Py_INCREF(ins); diff --git a/Objects/stringlib/string_format.h b/Objects/stringlib/string_format.h --- a/Objects/stringlib/string_format.h +++ b/Objects/stringlib/string_format.h @@ -209,19 +209,17 @@ if (digitval < 0) return -1; /* - This trick was copied from old Unicode format code. It's cute, - but would really suck on an old machine with a slow divide - implementation. Fortunately, in the normal case we do not - expect too many digits. + Detect possible overflow before it happens: + + accumulator * 10 + digitval > PY_SSIZE_T_MAX if and only if + accumulator > (PY_SSIZE_T_MAX - digitval) / 10. */ - oldaccumulator = accumulator; - accumulator *= 10; - if ((accumulator+10)/10 != oldaccumulator+1) { + if (accumulator > (PY_SSIZE_T_MAX - digitval) / 10) { PyErr_Format(PyExc_ValueError, "Too many decimal digits in format string"); return -1; } - accumulator += digitval; + accumulator = accumulator * 10 + digitval; } return accumulator; } -- Repository URL: http://hg.python.org/cpython [View Less]

1 0