Mailman 3 January 2012 - Python-Dev

io module types
by Matt Joiner Jan. 25, 2012

Jan. 25, 2012

Can calls to the C types in the io module be made into module lookups more akin to how it would work were it written in Python? The C implementation for io_open invokes the C type objects for FileIO, and friends, instead of looking them up on the io or _io modules. This makes it difficult to subclass and/or modify the behaviour of those classes from Python. http://hg.python.org/cpython/file/0bec943f6778/Modules/_io/_iomodule.c#l413

1 0

Re: [Python-Dev] Counting collisions w/ no need for a fatal exception
by Gregory P. Smith Jan. 25, 2012

Jan. 25, 2012

On Sun, Jan 22, 2012 at 10:41 PM, Tim Delaney <timothy.c.delaney(a)gmail.com> wrote: > On 23 January 2012 16:49, Lennart Regebro <regebro(a)gmail.com> wrote: >> >> On Mon, Jan 23, 2012 at 06:02, Paul McMillan <paul(a)mcmillan.ws> wrote: >> >> We may use a different salt per dictionary. >> > >> > If we're willing to re-hash everything on a per-dictionary basis. That >> > doesn't seem reasonable given our existing usage. >> >> Well, if we get crazy amounts of collisions, re-hashing with a new >> salt to get rid of those collisions seems quite reasonable to me... > > > Actually, this looks like it has the seed of a solution in it. I haven't > scrutinised the following beyond "it sounds like it could work" - it could > well contain nasty flaws. > > Assumption: We only get an excessive number of collisions during an attack > (directly or indirectly). > Assumption: Introducing a salt into hashes will change those hashes > sufficiently to mitigate the attack (all discussion of randomising hashes > makes this assumption). > > 1. Keep the current hashing (for all dictionaries) i.e. just using > hash(key). > > 2. Count collisions. > > 3. If any key hits X collisions change that dictionary to use a random salt > for hashes (at least for str and unicode keys). This salt would be > remembered for the dictionary. > > Consequence: The dictionary would need to be rebuilt when an attack was > detected. > Consequence: Hash caching would no longer occur for this dictionary, making > most operations more expensive. > Consequence: Anything relying on the iteration order of a dictionary which > has suffered excessive conflicts would fail. +1 I like this! The dictionary would still be O(n) but the constant cost in front of that just went up. When you are dealing with keys coming in from outside of the process, those are unlikely to already have any hash values so the constant cost at insertion time has really not changed at all because they would need hashing anyways. Their cost at non-iteration lookup time will be a constant factor greater but I do not see that as being a problem given that known keys being looked up in a This approach also allows for the dictionary hashing mode switch to occur after a lower number of collisions than the previous investigations into raising a MemoryError or similar were asking for (because they wanted to avoid false hard failures). It prevents that case from breaking in favor of a brief performance hiccup. I would *combine* this with a per process/interpreter-instance seed in 3.3 and later for added impact (less need for this code path to ever be triggered). For the purposes of backporting as a security fix, that part would be disabled by default but #1-3 would be enabled by default. Question A: Does the dictionary get rebuilt -again- with a new dict-salt if a large number of collisions occurs after a dict-salt has already been established? Question B: Is there a size of dictionary in which we refuse to rebuild & rehash it because it would simply be too costly? obviously if we lack the ram to malloc a new table, when else? ever? Suggestion: Would there be any benefit to making the number of collisions threshold on when to rebuild & rehash a log function of the dictionary's current size rather than a constant for all dicts? > > 4. (Optional) in 3.3, provide a way to get a dictionary with random salt > (i.e. not wait for attack). I don't like #4 as a documented public API as I'm not sure how well that'd play with other VMs (I suppose they could ignore it) but it would be useful for dict implementation testing purposes and easier studying of the behavior. If this is added it should be a method on the dict such as ._set_hash_salt() or something and for testing purposes it would be good to allow a dictionary to be queried to see if they are using their own salt or not (perhaps just ._get_hash_salt() returning non 0 means they are?) -gps

1 0

Re: [Python-Dev] devguide: Use -j0 to maximimze parallel execution.
by Georg Brandl Jan. 24, 2012

Jan. 24, 2012

Am 24.01.2012 18:58, schrieb brett.cannon: > http://hg.python.org/devguide/rev/a34e4a6b89dc > changeset: 489:a34e4a6b89dc > user: Brett Cannon <brett(a)python.org> > date: Tue Jan 24 12:58:01 2012 -0500 > summary: > Use -j0 to maximimze parallel execution. > > files: > runtests.rst | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/runtests.rst b/runtests.rst > --- a/runtests.rst > +++ b/runtests.rst > @@ -41,7 +41,7 @@ > If you have a multi-core or multi-CPU machine, you can enable parallel testing > using several Python processes so as to speed up things:: > > - ./python -m test -j2 > + ./python -m test -j0 That only works on 3.3 though... Georg

2 1

Sprinting at PyCon US
by Brett Cannon Jan. 24, 2012

Jan. 24, 2012

I went ahead and signed us up as usual: https://us.pycon.org/2012/community/sprints/projects/ . I listed myself as the leader, but I will only be at the sprints one full day and whatever part of Tuesday I can fit in before flying out to Toronto (which is probably not much thanks to the timezone difference). So if someone wants to be the official leader who will be there longer feel free to take me off and put yourself in (and you don't need to ask me beforehand).

1 0

exception chaining
by Ethan Furman Jan. 23, 2012

Jan. 23, 2012

Summary: Exception Chaining is cool, unless you are writing libraries that want to transform from Exception X to Exception Y as the the previous exception context is unnecessary, potentially confusing, and cluttery (yup, just made that word up!). For all the gory details, see http://bugs.python.org/issue6210. I'm going to attempt a patch implementing MRAB's suggestion: try: some_op except ValueError: raise as OtherError() # `raise` keeps context, `raise as` does not The question I have at the moment is: should `raise as` be an error if no exception is currently being handled? Example: def smurfy(x): if x != 'magic flute': raise as WrongInstrument do_something_with_x If this is allowed then `smurfy` could be called from inside an `except` clause or outside it. I don't care for it for two reasons: - I don't like the way it looks - I can see it encouraging always using `raise as` instead of `raise` and losing the value of exception chaining. Other thoughts? ~Ethan~

6 13

python build failed on mac
by Vijay N. Majagaonkar Jan. 22, 2012

Jan. 22, 2012

Hi all, I am trying to build python 3 on mac and build failing with following error can somebody help me with this $ hg clone http://hg.python.org/cpython $ ./configure $ make gcc -framework CoreFoundation -o python.exe Modules/python.o libpython3.3m.a -ldl -framework CoreFoundation ./python.exe -SE -m sysconfig --generate-posix-vars Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] python.exe(43296) malloc: *** mmap(size=7310873954244194304) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug make: *** [Lib/_sysconfigdata.py] Segmentation fault: 11 make: *** Deleting file `Lib/_sysconfigdata.py' ;)

5 7

Re: [Python-Dev] [Python-checkins] cpython (3.2): Fixes issue #8052: The posix subprocess module's close_fds behavior was
by Benjamin Peterson Jan. 22, 2012

Jan. 22, 2012

2012/1/21 gregory.p.smith <python-checkins(a)python.org>: ... > +/* Convert ASCII to a positive int, no libc call. no overflow. -1 on error. */ Is no libc call important? > +static int _pos_int_from_ascii(char *name) To be consistent with the rest of posixmodule.c, "static int" should be on a different line from the signature. This also applies to all other function declarations added by this. > +{ > + int num = 0; > + while (*name >= '0' && *name <= '9') { > + num = num * 10 + (*name - '0'); > + ++name; > + } > + if (*name) > + return -1; /* Non digit found, not a number. */ > + return num; > +} > + > + > +/* Returns 1 if there is a problem with fd_sequence, 0 otherwise. */ > +static int _sanity_check_python_fd_sequence(PyObject *fd_sequence) > +{ > + Py_ssize_t seq_idx, seq_len = PySequence_Length(fd_sequence); PySequence_Length can fail. > + long prev_fd = -1; > + for (seq_idx = 0; seq_idx < seq_len; ++seq_idx) { > + PyObject* py_fd = PySequence_Fast_GET_ITEM(fd_sequence, seq_idx); > + long iter_fd = PyLong_AsLong(py_fd); > + if (iter_fd < 0 || iter_fd < prev_fd || iter_fd > INT_MAX) { > + /* Negative, overflow, not a Long, unsorted, too big for a fd. */ > + return 1; > + } > + } > + return 0; > +} > + > + > +/* Is fd found in the sorted Python Sequence? */ > +static int _is_fd_in_sorted_fd_sequence(int fd, PyObject *fd_sequence) > +{ > + /* Binary search. */ > + Py_ssize_t search_min = 0; > + Py_ssize_t search_max = PySequence_Length(fd_sequence) - 1; > + if (search_max < 0) > + return 0; > + do { > + long middle = (search_min + search_max) / 2; > + long middle_fd = PyLong_AsLong( > + PySequence_Fast_GET_ITEM(fd_sequence, middle)); No check for error? > + if (fd == middle_fd) > + return 1; > + if (fd > middle_fd) > + search_min = middle + 1; > + else > + search_max = middle - 1; > + } while (search_min <= search_max); > + return 0; > +} > + > + > +/* Close all file descriptors in the range start_fd inclusive to > + * end_fd exclusive except for those in py_fds_to_keep. If the > + * range defined by [start_fd, end_fd) is large this will take a > + * long time as it calls close() on EVERY possible fd. > + */ > +static void _close_fds_by_brute_force(int start_fd, int end_fd, > + PyObject *py_fds_to_keep) > +{ > + Py_ssize_t num_fds_to_keep = PySequence_Length(py_fds_to_keep); > + Py_ssize_t keep_seq_idx; > + int fd_num; > + /* As py_fds_to_keep is sorted we can loop through the list closing > + * fds inbetween any in the keep list falling within our range. */ > + for (keep_seq_idx = 0; keep_seq_idx < num_fds_to_keep; ++keep_seq_idx) { > + PyObject* py_keep_fd = PySequence_Fast_GET_ITEM(py_fds_to_keep, > + keep_seq_idx); > + int keep_fd = PyLong_AsLong(py_keep_fd); > + if (keep_fd < start_fd) > + continue; > + for (fd_num = start_fd; fd_num < keep_fd; ++fd_num) { > + while (close(fd_num) < 0 && errno == EINTR); > + } > + start_fd = keep_fd + 1; > + } > + if (start_fd <= end_fd) { > + for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { > + while (close(fd_num) < 0 && errno == EINTR); > + } > + } > +} > + > + > +#if defined(__linux__) && defined(HAVE_SYS_SYSCALL_H) > +/* It doesn't matter if d_name has room for NAME_MAX chars; we're using this > + * only to read a directory of short file descriptor number names. The kernel > + * will return an error if we didn't give it enough space. Highly Unlikely. > + * This structure is very old and stable: It will not change unless the kernel > + * chooses to break compatibility with all existing binaries. Highly Unlikely. > + */ > +struct linux_dirent { > + unsigned long d_ino; /* Inode number */ > + unsigned long d_off; /* Offset to next linux_dirent */ > + unsigned short d_reclen; /* Length of this linux_dirent */ > + char d_name[256]; /* Filename (null-terminated) */ > +}; > + > +/* Close all open file descriptors in the range start_fd inclusive to end_fd > + * exclusive. Do not close any in the sorted py_fds_to_keep list. > + * > + * This version is async signal safe as it does not make any unsafe C library > + * calls, malloc calls or handle any locks. It is _unfortunate_ to be forced > + * to resort to making a kernel system call directly but this is the ONLY api > + * available that does no harm. opendir/readdir/closedir perform memory > + * allocation and locking so while they usually work they are not guaranteed > + * to (especially if you have replaced your malloc implementation). A version > + * of this function that uses those can be found in the _maybe_unsafe variant. > + * > + * This is Linux specific because that is all I am ready to test it on. It > + * should be easy to add OS specific dirent or dirent64 structures and modify > + * it with some cpp #define magic to work on other OSes as well if you want. > + */ > +static void _close_open_fd_range_safe(int start_fd, int end_fd, > + PyObject* py_fds_to_keep) > +{ > + int fd_dir_fd; > + if (start_fd >= end_fd) > + return; > + fd_dir_fd = open(LINUX_SOLARIS_FD_DIR, O_RDONLY | O_CLOEXEC, 0); > + /* Not trying to open the BSD_OSX path as this is currently Linux only. */ > + if (fd_dir_fd == -1) { > + /* No way to get a list of open fds. */ > + _close_fds_by_brute_force(start_fd, end_fd, py_fds_to_keep); > + return; > + } else { > + char buffer[sizeof(struct linux_dirent)]; > + int bytes; > + while ((bytes = syscall(SYS_getdents, fd_dir_fd, > + (struct linux_dirent *)buffer, > + sizeof(buffer))) > 0) { > + struct linux_dirent *entry; > + int offset; > + for (offset = 0; offset < bytes; offset += entry->d_reclen) { > + int fd; > + entry = (struct linux_dirent *)(buffer + offset); > + if ((fd = _pos_int_from_ascii(entry->d_name)) < 0) > + continue; /* Not a number. */ > + if (fd != fd_dir_fd && fd >= start_fd && fd < end_fd && > + !_is_fd_in_sorted_fd_sequence(fd, py_fds_to_keep)) { > + while (close(fd) < 0 && errno == EINTR); > + } > + } > + } > + close(fd_dir_fd); > + } > +} > + > +#define _close_open_fd_range _close_open_fd_range_safe > + > +#else /* NOT (defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)) */ > + > + > +/* Close all open file descriptors in the range start_fd inclusive to end_fd > + * exclusive. Do not close any in the sorted py_fds_to_keep list. > + * > + * This function violates the strict use of async signal safe functions. :( > + * It calls opendir(), readdir64() and closedir(). Of these, the one most > + * likely to ever cause a problem is opendir() as it performs an internal > + * malloc(). Practically this should not be a problem. The Java VM makes the > + * same calls between fork and exec in its own UNIXProcess_md.c implementation. > + * > + * readdir_r() is not used because it provides no benefit. It is typically > + * implemented as readdir() followed by memcpy(). See also: > + * http://womble.decadent.org.uk/readdir_r-advisory.html > + */ > +static void _close_open_fd_range_maybe_unsafe(int start_fd, int end_fd, > + PyObject* py_fds_to_keep) > +{ > + DIR *proc_fd_dir; > +#ifndef HAVE_DIRFD > + while (_is_fd_in_sorted_fd_sequence(start_fd, py_fds_to_keep) && > + (start_fd < end_fd)) { > + ++start_fd; > + } > + if (start_fd >= end_fd) > + return; > + /* Close our lowest fd before we call opendir so that it is likely to > + * reuse that fd otherwise we might close opendir's file descriptor in > + * our loop. This trick assumes that fd's are allocated on a lowest > + * available basis. */ > + while (close(start_fd) < 0 && errno == EINTR); > + ++start_fd; > +#endif > + if (start_fd >= end_fd) > + return; > + > + proc_fd_dir = opendir(BSD_OSX_FD_DIR); > + if (!proc_fd_dir) > + proc_fd_dir = opendir(LINUX_SOLARIS_FD_DIR); > + if (!proc_fd_dir) { > + /* No way to get a list of open fds. */ > + _close_fds_by_brute_force(start_fd, end_fd, py_fds_to_keep); > + } else { > + struct dirent64 *dir_entry; > +#ifdef HAVE_DIRFD > + int fd_used_by_opendir = DIRFD(proc_fd_dir); > +#else > + int fd_used_by_opendir = start_fd - 1; > +#endif > + errno = 0; > + /* readdir64 is used to work around Solaris 9 bug 6395699. */ > + while ((dir_entry = readdir64(proc_fd_dir))) { > + int fd; > + if ((fd = _pos_int_from_ascii(dir_entry->d_name)) < 0) > + continue; /* Not a number. */ > + if (fd != fd_used_by_opendir && fd >= start_fd && fd < end_fd && > + !_is_fd_in_sorted_fd_sequence(fd, py_fds_to_keep)) { > + while (close(fd) < 0 && errno == EINTR); > + } > + errno = 0; > + } > + if (errno) { > + /* readdir error, revert behavior. Highly Unlikely. */ > + _close_fds_by_brute_force(start_fd, end_fd, py_fds_to_keep); > + } > + closedir(proc_fd_dir); > + } > +} > + > +#define _close_open_fd_range _close_open_fd_range_maybe_unsafe > + > +#endif /* else NOT (defined(__linux__) && defined(HAVE_SYS_SYSCALL_H)) */ > + > + > /* > * This function is code executed in the child process immediately after fork > * to set things up and call exec(). > @@ -46,12 +292,12 @@ > int errread, int errwrite, > int errpipe_read, int errpipe_write, > int close_fds, int restore_signals, > - int call_setsid, Py_ssize_t num_fds_to_keep, > + int call_setsid, > PyObject *py_fds_to_keep, > PyObject *preexec_fn, > PyObject *preexec_fn_args_tuple) > { > - int i, saved_errno, fd_num, unused; > + int i, saved_errno, unused; > PyObject *result; > const char* err_msg = ""; > /* Buffer large enough to hold a hex integer. We can't malloc. */ > @@ -113,33 +359,8 @@ > POSIX_CALL(close(errwrite)); > } > > - /* close() is intentionally not checked for errors here as we are closing */ > - /* a large range of fds, some of which may be invalid. */ > - if (close_fds) { > - Py_ssize_t keep_seq_idx; > - int start_fd = 3; > - for (keep_seq_idx = 0; keep_seq_idx < num_fds_to_keep; ++keep_seq_idx) { > - PyObject* py_keep_fd = PySequence_Fast_GET_ITEM(py_fds_to_keep, > - keep_seq_idx); > - int keep_fd = PyLong_AsLong(py_keep_fd); > - if (keep_fd < 0) { /* Negative number, overflow or not a Long. */ > - err_msg = "bad value in fds_to_keep."; > - errno = 0; /* We don't want to report an OSError. */ > - goto error; > - } > - if (keep_fd < start_fd) > - continue; > - for (fd_num = start_fd; fd_num < keep_fd; ++fd_num) { > - close(fd_num); > - } > - start_fd = keep_fd + 1; > - } > - if (start_fd <= max_fd) { > - for (fd_num = start_fd; fd_num < max_fd; ++fd_num) { > - close(fd_num); > - } > - } > - } > + if (close_fds) > + _close_open_fd_range(3, max_fd, py_fds_to_keep); > > if (cwd) > POSIX_CALL(chdir(cwd)); > @@ -227,7 +448,7 @@ > pid_t pid; > int need_to_reenable_gc = 0; > char *const *exec_array, *const *argv = NULL, *const *envp = NULL; > - Py_ssize_t arg_num, num_fds_to_keep; > + Py_ssize_t arg_num; > > if (!PyArg_ParseTuple( > args, "OOOOOOiiiiiiiiiiO:fork_exec", > @@ -243,9 +464,12 @@ > PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); > return NULL; > } > - num_fds_to_keep = PySequence_Length(py_fds_to_keep); > - if (num_fds_to_keep < 0) { > - PyErr_SetString(PyExc_ValueError, "bad fds_to_keep"); > + if (PySequence_Length(py_fds_to_keep) < 0) { > + PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); > + return NULL; > + } > + if (_sanity_check_python_fd_sequence(py_fds_to_keep)) { > + PyErr_SetString(PyExc_ValueError, "bad value(s) in fds_to_keep"); > return NULL; > } > > @@ -348,8 +572,7 @@ > p2cread, p2cwrite, c2pread, c2pwrite, > errread, errwrite, errpipe_read, errpipe_write, > close_fds, restore_signals, call_setsid, > - num_fds_to_keep, py_fds_to_keep, > - preexec_fn, preexec_fn_args_tuple); > + py_fds_to_keep, preexec_fn, preexec_fn_args_tuple); > _exit(255); > return NULL; /* Dead code to avoid a potential compiler warning. */ > } > diff --git a/configure b/configure > --- a/configure > +++ b/configure > @@ -6165,7 +6165,7 @@ > sys/audioio.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \ > sys/lock.h sys/mkdev.h sys/modem.h \ > sys/param.h sys/poll.h sys/select.h sys/socket.h sys/statvfs.h sys/stat.h \ > -sys/termio.h sys/time.h \ > +sys/syscall.h sys/termio.h sys/time.h \ > sys/times.h sys/types.h sys/un.h sys/utsname.h sys/wait.h pty.h libutil.h \ > sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \ > bluetooth/bluetooth.h linux/tipc.h spawn.h util.h > diff --git a/configure.in b/configure.in > --- a/configure.in > +++ b/configure.in > @@ -1341,7 +1341,7 @@ > sys/audioio.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \ > sys/lock.h sys/mkdev.h sys/modem.h \ > sys/param.h sys/poll.h sys/select.h sys/socket.h sys/statvfs.h sys/stat.h \ > -sys/termio.h sys/time.h \ > +sys/syscall.h sys/termio.h sys/time.h \ > sys/times.h sys/types.h sys/un.h sys/utsname.h sys/wait.h pty.h libutil.h \ > sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \ > bluetooth/bluetooth.h linux/tipc.h spawn.h util.h) > diff --git a/pyconfig.h.in b/pyconfig.h.in > --- a/pyconfig.h.in > +++ b/pyconfig.h.in > @@ -789,6 +789,9 @@ > /* Define to 1 if you have the <sys/stat.h> header file. */ > #undef HAVE_SYS_STAT_H > > +/* Define to 1 if you have the <sys/syscall.h> header file. */ > +#undef HAVE_SYS_SYSCALL_H > + > /* Define to 1 if you have the <sys/termio.h> header file. */ > #undef HAVE_SYS_TERMIO_H -- Regards, Benjamin

2 1

Re: [Python-Dev] cpython (3.2): Avoid the compiler warning about the unused return value.
by Antoine Pitrou Jan. 22, 2012

Jan. 22, 2012

On Sat, 21 Jan 2012 21:51:43 +0100 gregory.p.smith <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/d01fecadf3ea > changeset: 74561:d01fecadf3ea > branch: 3.2 > parent: 74558:03e61104f7a2 > user: Gregory P. Smith <greg(a)krypto.org> > date: Sat Jan 21 12:31:25 2012 -0800 > summary: > Avoid the compiler warning about the unused return value. Can't that give you another warning about the ssize_t being truncated to int? How about the following instead? (void) write(...); cheers Antoine.

5 6

Re: [Python-Dev] cpython (3.2): Fixes issue #8052: The posix subprocess module's close_fds behavior was
by Antoine Pitrou Jan. 21, 2012

Jan. 21, 2012

On Sat, 21 Jan 2012 23:39:41 +0100 gregory.p.smith <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/61aa484a3e54 > changeset: 74563:61aa484a3e54 > branch: 3.2 > parent: 74561:d01fecadf3ea > user: Gregory P. Smith <greg(a)krypto.org> > date: Sat Jan 21 14:01:08 2012 -0800 > summary: > Fixes issue #8052: The posix subprocess module's close_fds behavior was > suboptimal by closing all possible file descriptors rather than just > the open ones in the child process before exec(). For what it's worth, I'm not really confident with so much new low-level code in a bugfix release. IMHO it's more of a new feature, since it's a performance improvement. Regards Antoine.

2 1

2.7 now uses Sphinx 1.0
by Sandro Tosi Jan. 21, 2012

Jan. 21, 2012

Hello, just a heads-up: documentation for 2.7 branch has been ported to use sphinx 1.0, so now the same syntax can be used for 2.x and 3.x patches, hopefully easying working on both python stacks. Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi

4 3