Mailman 3 March 2012 - Python-Dev

im_func: implementation detail?
by Ethan Furman 11 Mar '12

11 Mar '12

How does someone know if something in CPython is an implementation detail or not? In the case of im_func, I think it is (an implementation detail), and another person thinks it is part of the language spec. That all implementations must have. ~Ethan~

2 1

Re: [Python-Dev] [Python-checkins] peps: PEP 416: remove mentions of mutable/immutable
by Terry Reedy 10 Mar '12

10 Mar '12

On 3/10/2012 5:43 AM, victor.stinner wrote: > http://hg.python.org/peps/rev/7278026a5db9 > changeset: 4124:7278026a5db9 > user: Victor Stinner<victor.stinner(a)gmail.com> > date: Sat Mar 10 11:43:45 2012 +0100 > summary: > PEP 416: remove mentions of mutable/immutable > > files: > pep-0416.txt | 5 ++--- > 1 files changed, 2 insertions(+), 3 deletions(-) > > > diff --git a/pep-0416.txt b/pep-0416.txt > --- a/pep-0416.txt > +++ b/pep-0416.txt > @@ -20,9 +20,8 @@ > ========= > > A frozendict is a read-only mapping: a key cannot be added nor removed, and a > -key is always mapped to the same value. However, frozendict values can be > -mutable (not hashable). A frozendict is hashable and so immutable if and only > -if all values are hashable (immutable). > +key is always mapped to the same value. However, frozendict values can be not > +hashable. A frozendict is hashable if and only if all values are hashable. I think this would be better with 'key is always mapped to the same value object'. The revised second sentence does not add anything. Some Python objects are hashable and some not. There is nothing special about frozendict value objects either way. The proper revision of your original is "However, frozendict value may not be hashable." Either change to that or remove it. Terry

1 0

Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
by Jim Jewett 10 Mar '12

10 Mar '12

I do not believe the change set below is valid. As I read it, the new test verifies that one particular type of Nasty key will provoke a RuntimeError -- but that particular type already did so, by hitting the recursion limit. (It doesn't even really mutate the dict.) Meanwhile, the patch throws out tests for several different types of mutations that have caused problems -- even segfaults -- in the past, even after the dict implementation code was already "fixed". Changing these tests to "assertRaises" would be fine, but they should all be kept; if nothing else, they test whether you've caught all mutation avenues. -jJ On Mon, Mar 5, 2012 at 7:13 PM, victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/934aaf2191d0 > changeset: 75445:934aaf2191d0 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Mar 06 01:03:13 2012 +0100 > summary: > Close #14205: dict lookup raises a RuntimeError if the dict is modified during > a lookup. > > "if you want to make a sandbox on top of CPython, you have to fix segfaults" > so let's fix segfaults! > > files: > Lib/test/crashers/nasty_eq_vs_dict.py | 47 -- > Lib/test/test_dict.py | 22 +- > Lib/test/test_mutants.py | 291 -------------- > Misc/NEWS | 5 +- > Objects/dictobject.c | 18 +- > 5 files changed, 31 insertions(+), 352 deletions(-) > > > diff --git a/Lib/test/crashers/nasty_eq_vs_dict.py b/Lib/test/crashers/nasty_eq_vs_dict.py > deleted file mode 100644 > --- a/Lib/test/crashers/nasty_eq_vs_dict.py > +++ /dev/null > @@ -1,47 +0,0 @@ > -# from http://mail.python.org/pipermail/python-dev/2001-June/015239.html > - > -# if you keep changing a dictionary while looking up a key, you can > -# provoke an infinite recursion in C > - > -# At the time neither Tim nor Michael could be bothered to think of a > -# way to fix it. > - > -class Yuck: > - def __init__(self): > - self.i = 0 > - > - def make_dangerous(self): > - self.i = 1 > - > - def __hash__(self): > - # direct to slot 4 in table of size 8; slot 12 when size 16 > - return 4 + 8 > - > - def __eq__(self, other): > - if self.i == 0: > - # leave dict alone > - pass > - elif self.i == 1: > - # fiddle to 16 slots > - self.__fill_dict(6) > - self.i = 2 > - else: > - # fiddle to 8 slots > - self.__fill_dict(4) > - self.i = 1 > - > - return 1 > - > - def __fill_dict(self, n): > - self.i = 0 > - dict.clear() > - for i in range(n): > - dict[i] = i > - dict[self] = "OK!" > - > -y = Yuck() > -dict = {y: "OK!"} > - > -z = Yuck() > -y.make_dangerous() > -print(dict[z]) > diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py > --- a/Lib/test/test_dict.py > +++ b/Lib/test/test_dict.py > @@ -379,7 +379,7 @@ > x.fail = True > self.assertRaises(Exc, d.pop, x) > > - def test_mutatingiteration(self): > + def test_mutating_iteration(self): > # changing dict size during iteration > d = {} > d[1] = 1 > @@ -387,6 +387,26 @@ > for i in d: > d[i+1] = 1 > > + def test_mutating_lookup(self): > + # changing dict during a lookup > + class NastyKey: > + mutate_dict = None > + > + def __hash__(self): > + # hash collision! > + return 1 > + > + def __eq__(self, other): > + if self.mutate_dict: > + self.mutate_dict[self] = 1 > + return self == other > + > + d = {} > + d[NastyKey()] = 0 > + NastyKey.mutate_dict = d > + with self.assertRaises(RuntimeError): > + d[NastyKey()] = None > + > def test_repr(self): > d = {} > self.assertEqual(repr(d), '{}') > diff --git a/Lib/test/test_mutants.py b/Lib/test/test_mutants.py > deleted file mode 100644 > --- a/Lib/test/test_mutants.py > +++ /dev/null > @@ -1,291 +0,0 @@ > -from test.support import verbose, TESTFN > -import random > -import os > - > -# From SF bug #422121: Insecurities in dict comparison. > - > -# Safety of code doing comparisons has been an historical Python weak spot. > -# The problem is that comparison of structures written in C *naturally* > -# wants to hold on to things like the size of the container, or "the > -# biggest" containee so far, across a traversal of the container; but > -# code to do containee comparisons can call back into Python and mutate > -# the container in arbitrary ways while the C loop is in midstream. If the > -# C code isn't extremely paranoid about digging things out of memory on > -# each trip, and artificially boosting refcounts for the duration, anything > -# from infinite loops to OS crashes can result (yes, I use Windows <wink>). > -# > -# The other problem is that code designed to provoke a weakness is usually > -# white-box code, and so catches only the particular vulnerabilities the > -# author knew to protect against. For example, Python's list.sort() code > -# went thru many iterations as one "new" vulnerability after another was > -# discovered. > -# > -# So the dict comparison test here uses a black-box approach instead, > -# generating dicts of various sizes at random, and performing random > -# mutations on them at random times. This proved very effective, > -# triggering at least six distinct failure modes the first 20 times I > -# ran it. Indeed, at the start, the driver never got beyond 6 iterations > -# before the test died. > - > -# The dicts are global to make it easy to mutate tham from within functions. > -dict1 = {} > -dict2 = {} > - > -# The current set of keys in dict1 and dict2. These are materialized as > -# lists to make it easy to pick a dict key at random. > -dict1keys = [] > -dict2keys = [] > - > -# Global flag telling maybe_mutate() whether to *consider* mutating. > -mutate = 0 > - > -# If global mutate is true, consider mutating a dict. May or may not > -# mutate a dict even if mutate is true. If it does decide to mutate a > -# dict, it picks one of {dict1, dict2} at random, and deletes a random > -# entry from it; or, more rarely, adds a random element. > - > -def maybe_mutate(): > - global mutate > - if not mutate: > - return > - if random.random() < 0.5: > - return > - > - if random.random() < 0.5: > - target, keys = dict1, dict1keys > - else: > - target, keys = dict2, dict2keys > - > - if random.random() < 0.2: > - # Insert a new key. > - mutate = 0 # disable mutation until key inserted > - while 1: > - newkey = Horrid(random.randrange(100)) > - if newkey not in target: > - break > - target[newkey] = Horrid(random.randrange(100)) > - keys.append(newkey) > - mutate = 1 > - > - elif keys: > - # Delete a key at random. > - mutate = 0 # disable mutation until key deleted > - i = random.randrange(len(keys)) > - key = keys[i] > - del target[key] > - del keys[i] > - mutate = 1 > - > -# A horrid class that triggers random mutations of dict1 and dict2 when > -# instances are compared. > - > -class Horrid: > - def __init__(self, i): > - # Comparison outcomes are determined by the value of i. > - self.i = i > - > - # An artificial hashcode is selected at random so that we don't > - # have any systematic relationship between comparison outcomes > - # (based on self.i and other.i) and relative position within the > - # hash vector (based on hashcode). > - # XXX This is no longer effective. > - ##self.hashcode = random.randrange(1000000000) > - > - def __hash__(self): > - return 42 > - return self.hashcode > - > - def __eq__(self, other): > - maybe_mutate() # The point of the test. > - return self.i == other.i > - > - def __ne__(self, other): > - raise RuntimeError("I didn't expect some kind of Spanish inquisition!") > - > - __lt__ = __le__ = __gt__ = __ge__ = __ne__ > - > - def __repr__(self): > - return "Horrid(%d)" % self.i > - > -# Fill dict d with numentries (Horrid(i), Horrid(j)) key-value pairs, > -# where i and j are selected at random from the candidates list. > -# Return d.keys() after filling. > - > -def fill_dict(d, candidates, numentries): > - d.clear() > - for i in range(numentries): > - d[Horrid(random.choice(candidates))] = \ > - Horrid(random.choice(candidates)) > - return list(d.keys()) > - > -# Test one pair of randomly generated dicts, each with n entries. > -# Note that dict comparison is trivial if they don't have the same number > -# of entires (then the "shorter" dict is instantly considered to be the > -# smaller one, without even looking at the entries). > - > -def test_one(n): > - global mutate, dict1, dict2, dict1keys, dict2keys > - > - # Fill the dicts without mutating them. > - mutate = 0 > - dict1keys = fill_dict(dict1, range(n), n) > - dict2keys = fill_dict(dict2, range(n), n) > - > - # Enable mutation, then compare the dicts so long as they have the > - # same size. > - mutate = 1 > - if verbose: > - print("trying w/ lengths", len(dict1), len(dict2), end=' ') > - while dict1 and len(dict1) == len(dict2): > - if verbose: > - print(".", end=' ') > - c = dict1 == dict2 > - if verbose: > - print() > - > -# Run test_one n times. At the start (before the bugs were fixed), 20 > -# consecutive runs of this test each blew up on or before the sixth time > -# test_one was run. So n doesn't have to be large to get an interesting > -# test. > -# OTOH, calling with large n is also interesting, to ensure that the fixed > -# code doesn't hold on to refcounts *too* long (in which case memory would > -# leak). > - > -def test(n): > - for i in range(n): > - test_one(random.randrange(1, 100)) > - > -# See last comment block for clues about good values for n. > -test(100) > - > -########################################################################## > -# Another segfault bug, distilled by Michael Hudson from a c.l.py post. > - > -class Child: > - def __init__(self, parent): > - self.__dict__['parent'] = parent > - def __getattr__(self, attr): > - self.parent.a = 1 > - self.parent.b = 1 > - self.parent.c = 1 > - self.parent.d = 1 > - self.parent.e = 1 > - self.parent.f = 1 > - self.parent.g = 1 > - self.parent.h = 1 > - self.parent.i = 1 > - return getattr(self.parent, attr) > - > -class Parent: > - def __init__(self): > - self.a = Child(self) > - > -# Hard to say what this will print! May vary from time to time. But > -# we're specifically trying to test the tp_print slot here, and this is > -# the clearest way to do it. We print the result to a temp file so that > -# the expected-output file doesn't need to change. > - > -f = open(TESTFN, "w") > -print(Parent().__dict__, file=f) > -f.close() > -os.unlink(TESTFN) > - > -########################################################################## > -# And another core-dumper from Michael Hudson. > - > -dict = {} > - > -# Force dict to malloc its table. > -for i in range(1, 10): > - dict[i] = i > - > -f = open(TESTFN, "w") > - > -class Machiavelli: > - def __repr__(self): > - dict.clear() > - > - # Michael sez: "doesn't crash without this. don't know why." > - # Tim sez: "luck of the draw; crashes with or without for me." > - print(file=f) > - > - return repr("machiavelli") > - > - def __hash__(self): > - return 0 > - > -dict[Machiavelli()] = Machiavelli() > - > -print(str(dict), file=f) > -f.close() > -os.unlink(TESTFN) > -del f, dict > - > - > -########################################################################## > -# And another core-dumper from Michael Hudson. > - > -dict = {} > - > -# let's force dict to malloc its table > -for i in range(1, 10): > - dict[i] = i > - > -class Machiavelli2: > - def __eq__(self, other): > - dict.clear() > - return 1 > - > - def __hash__(self): > - return 0 > - > -dict[Machiavelli2()] = Machiavelli2() > - > -try: > - dict[Machiavelli2()] > -except KeyError: > - pass > - > -del dict > - > -########################################################################## > -# And another core-dumper from Michael Hudson. > - > -dict = {} > - > -# let's force dict to malloc its table > -for i in range(1, 10): > - dict[i] = i > - > -class Machiavelli3: > - def __init__(self, id): > - self.id = id > - > - def __eq__(self, other): > - if self.id == other.id: > - dict.clear() > - return 1 > - else: > - return 0 > - > - def __repr__(self): > - return "%s(%s)"%(self.__class__.__name__, self.id) > - > - def __hash__(self): > - return 0 > - > -dict[Machiavelli3(1)] = Machiavelli3(0) > -dict[Machiavelli3(2)] = Machiavelli3(0) > - > -f = open(TESTFN, "w") > -try: > - try: > - print(dict[Machiavelli3(2)], file=f) > - except KeyError: > - pass > -finally: > - f.close() > - os.unlink(TESTFN) > - > -del dict > -del dict1, dict2, dict1keys, dict2keys > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,10 +10,13 @@ > Core and Builtins > ----------------- > > +- Issue #14205: dict lookup raises a RuntimeError if the dict is modified > + during a lookup. > + > Library > ------- > > -- Issue #14168: Check for presence of Element._attrs in minidom before > +- Issue #14168: Check for presence of Element._attrs in minidom before > accessing it. > > - Issue #12328: Fix multiprocessing's use of overlapped I/O on Windows. > diff --git a/Objects/dictobject.c b/Objects/dictobject.c > --- a/Objects/dictobject.c > +++ b/Objects/dictobject.c > @@ -347,12 +347,9 @@ > return ep; > } > else { > - /* The compare did major nasty stuff to the > - * dict: start over. > - * XXX A clever adversary could prevent this > - * XXX from terminating. > - */ > - return lookdict(mp, key, hash); > + PyErr_SetString(PyExc_RuntimeError, > + "dictionary changed size during lookup"); > + return NULL; > } > } > freeslot = NULL; > @@ -379,12 +376,9 @@ > return ep; > } > else { > - /* The compare did major nasty stuff to the > - * dict: start over. > - * XXX A clever adversary could prevent this > - * XXX from terminating. > - */ > - return lookdict(mp, key, hash); > + PyErr_SetString(PyExc_RuntimeError, > + "dictionary changed size during lookup"); > + return NULL; > } > } > else if (ep->me_key == dummy && freeslot == NULL) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

6 5

Testsuite dependency on _testcapi
by Thomas Wouters 09 Mar '12

09 Mar '12

While testing Python 2.7 internally (at Google) I noticed that (now that ImportErrors aren't automatically test skips) lots of tests fail if you don't have the _testcapi module. These tests are (as far as I've seen) properly marked as cpython-only, but when some wacko decides the _testcapi module shouldn't, for example, be shipped to a million machines[*] that are never going to use it, it would be nice to still run the tests that can be run without _testcapi. Any objections to fixing the tests to use test.support.import_module() for _testcapi and a 'needs_testcapi' skipping decorator? To elaborate, we are also not shipping a couple of other modules (like distutils), but it's not unreasonable to expect those to exist (we modify the testsuite for that in our own builds only, instead, as well as making all our code deal with it.) The _testcapi module, however, is internal *and* meant for tests only, and used in quite a few tests (sometimes only in a single testfunction.) [*] 'a million machines' is not the actual number -- I don't know the actual number (but I'm sure it's bigger than that), I'm just tossing out some large number. -- Thomas Wouters <thomas(a)python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

4 3

problem with recursive "yield from" delegation
by Stefan Behnel 09 Mar '12

09 Mar '12

Hi, I found a problem in the current "yield from" implementation that I think is worth discussing: http://bugs.python.org/issue14220 Test code: def g1(): yield "y1" yield from g2() yield "y4" def g2(): yield "y2" try: yield from gi except ValueError: pass # catch "already running" error yield "y3" gi = g1() for y in gi: print("Yielded: %s" % (y,)) This is what it currently does: 1) g1() delegates to a new g2(), propagates its "y2" value and asks for the next value 2) g2 delegates back to the g1 instance and asks for its next value 3) Python sees the active delegation in g1 and asks g2 for its next value 4) g2 sees that it's already running and throws an exception Ok so far. Now: 5) the exception is propagated into g1 at call level 3) instead of the original requestor g2 one level above 6) g1 undelegates and terminates by the exception 7) g2 catches the exception, yields "y3" and then terminates normally 8) g1 gets control back but has already terminated and does nothing Effect: g1 does not yield "y4" anymore. The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2. I ran into this while trying to adapt the implementation for Cython, which has a different generator type implementation but otherwise uses more or less the same code now. But I'm not sure how to fix this one without major changes to the implementation, especially not without special casing the generator type on delegation (which won't work because CPython doesn't know about Cython generators). Any ideas? Stefan

6 16

Summary of Python tracker Issues
by Python tracker 09 Mar '12

09 Mar '12

ACTIVITY SUMMARY (2012-03-02 - 2012-03-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3318 (+19) closed 22704 (+44) total 26022 (+63) Open issues with patches: 1409 Issues opened (45) ================== #14179: Test coverage for lib/re.py http://bugs.python.org/issue14179 opened by flomm #14180: Factorize code to convert int/float to time_t, timeval or time http://bugs.python.org/issue14180 opened by haypo #14182: collections.Counter equality test thrown-off by zero counts http://bugs.python.org/issue14182 opened by rhettinger #14183: Test coverage for packaging.install and packaging.pypi.wrapper http://bugs.python.org/issue14183 opened by francismb #14184: test_recursion_limit fails on OS X when compiled with clang http://bugs.python.org/issue14184 opened by dk #14185: Failure to build _dbm with ndbm on Arch Linux http://bugs.python.org/issue14185 opened by anikom15 #14186: Link to PEP 3107 in "def" part of Language Reference http://bugs.python.org/issue14186 opened by cvrebert #14187: add "annotation" entry to Glossary http://bugs.python.org/issue14187 opened by cvrebert #14189: Documentation for some C APIs is missing clear specification o http://bugs.python.org/issue14189 opened by baruch.sterin #14190: Minor C API documentation bugs http://bugs.python.org/issue14190 opened by baruch.sterin #14191: argparse: nargs='*' doesn't get out-of-order positional parame http://bugs.python.org/issue14191 opened by v+python #14196: Unhandled exceptions in pdb return value display http://bugs.python.org/issue14196 opened by Simon.Chopin #14197: OS X framework builds do not create ABI-suffixed libpython3.x http://bugs.python.org/issue14197 opened by shibingli #14198: Backport parts of the new memoryview documentation http://bugs.python.org/issue14198 opened by skrah #14200: Idle shell crash on printing non-BMP unicode character http://bugs.python.org/issue14200 opened by vbr #14201: Documented caching for shared library's __getattr__ and __geti http://bugs.python.org/issue14201 opened by erijo #14202: The docs of xml.dom.pulldom are almost nonexistent http://bugs.python.org/issue14202 opened by eli.bendersky #14203: bytearray_getbuffer: unnecessary code http://bugs.python.org/issue14203 opened by skrah #14204: Support for the NPN extension to TLS/SSL http://bugs.python.org/issue14204 opened by colinmarc #14206: multiprocessing.Queue documentation is lacking important detai http://bugs.python.org/issue14206 opened by Garrett.Moore #14207: ElementTree.ParseError - needs documentation and consistent C& http://bugs.python.org/issue14207 opened by eli.bendersky #14208: No way to recover original argv with python -m http://bugs.python.org/issue14208 opened by Ben.Darnell #14209: pkgutil.iter_zipimport_modules ignores the prefix parameter fo http://bugs.python.org/issue14209 opened by James.Pickering #14210: add filename completion to pdb http://bugs.python.org/issue14210 opened by tshepang #14214: test_concurrent_futures hangs http://bugs.python.org/issue14214 opened by tebeka #14215: http://www.python.org/dev/peps/ title is python.org http://bugs.python.org/issue14215 opened by ramchandra.apte #14216: ImportError: No module named binascii http://bugs.python.org/issue14216 opened by liuqianhn #14219: start the Class tutorial in a more gentle manner http://bugs.python.org/issue14219 opened by tshepang #14222: Using time.time() in Queue.get breaks when system time is chan http://bugs.python.org/issue14222 opened by tasslehoff #14224: packaging: path description of resources is mixed up http://bugs.python.org/issue14224 opened by jokoala #14225: _cursesmodule compile error in OS X 32-bit-only installer buil http://bugs.python.org/issue14225 opened by ned.deily #14226: Expose dict_proxy type from descrobject.c http://bugs.python.org/issue14226 opened by ndparker #14227: console w/ cp65001 displays extra characters for non-ascii str http://bugs.python.org/issue14227 opened by metolone #14228: It is impossible to catch sigint on startup in python code http://bugs.python.org/issue14228 opened by telmich #14229: On KeyboardInterrupt, the exit code should mirror the signal n http://bugs.python.org/issue14229 opened by pitrou #14230: Delegating generator is not always visible to debugging tools http://bugs.python.org/issue14230 opened by Mark.Shannon #14232: obmalloc: mmap() returns MAP_FAILED on error, not 0 http://bugs.python.org/issue14232 opened by haypo #14234: CVE-2012-0876 (hash table collisions CPU usage DoS) for embedd http://bugs.python.org/issue14234 opened by dmalcolm #14235: test_cmd.py does not correctly call reload() http://bugs.python.org/issue14235 opened by Josh.Watson #14236: In help(re) are insufficient details http://bugs.python.org/issue14236 opened by py.user #14237: Special sequences \A and \Z don't work in character set [] http://bugs.python.org/issue14237 opened by py.user #14238: python shouldn't need username in passwd database http://bugs.python.org/issue14238 opened by arekm #14239: Uninitialised variable in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue14239 opened by Mark.Shannon #14241: io.UnsupportedOperation.__new__(io.UnsupportedOperation) fails http://bugs.python.org/issue14241 opened by Mark.Shannon #14218: include rendered output in addition to markup http://bugs.python.org/issue14218 opened by tshepang Most recent 15 issues with no replies (15) ========================================== #14241: io.UnsupportedOperation.__new__(io.UnsupportedOperation) fails http://bugs.python.org/issue14241 #14239: Uninitialised variable in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue14239 #14236: In help(re) are insufficient details http://bugs.python.org/issue14236 #14230: Delegating generator is not always visible to debugging tools http://bugs.python.org/issue14230 #14226: Expose dict_proxy type from descrobject.c http://bugs.python.org/issue14226 #14225: _cursesmodule compile error in OS X 32-bit-only installer buil http://bugs.python.org/issue14225 #14215: http://www.python.org/dev/peps/ title is python.org http://bugs.python.org/issue14215 #14207: ElementTree.ParseError - needs documentation and consistent C& http://bugs.python.org/issue14207 #14198: Backport parts of the new memoryview documentation http://bugs.python.org/issue14198 #14196: Unhandled exceptions in pdb return value display http://bugs.python.org/issue14196 #14189: Documentation for some C APIs is missing clear specification o http://bugs.python.org/issue14189 #14186: Link to PEP 3107 in "def" part of Language Reference http://bugs.python.org/issue14186 #14185: Failure to build _dbm with ndbm on Arch Linux http://bugs.python.org/issue14185 #14184: test_recursion_limit fails on OS X when compiled with clang http://bugs.python.org/issue14184 #14182: collections.Counter equality test thrown-off by zero counts http://bugs.python.org/issue14182 Most recent 15 issues waiting for review (15) ============================================= #14241: io.UnsupportedOperation.__new__(io.UnsupportedOperation) fails http://bugs.python.org/issue14241 #14239: Uninitialised variable in _PyObject_GenericSetAttrWithDict http://bugs.python.org/issue14239 #14235: test_cmd.py does not correctly call reload() http://bugs.python.org/issue14235 #14234: CVE-2012-0876 (hash table collisions CPU usage DoS) for embedd http://bugs.python.org/issue14234 #14232: obmalloc: mmap() returns MAP_FAILED on error, not 0 http://bugs.python.org/issue14232 #14230: Delegating generator is not always visible to debugging tools http://bugs.python.org/issue14230 #14224: packaging: path description of resources is mixed up http://bugs.python.org/issue14224 #14210: add filename completion to pdb http://bugs.python.org/issue14210 #14204: Support for the NPN extension to TLS/SSL http://bugs.python.org/issue14204 #14203: bytearray_getbuffer: unnecessary code http://bugs.python.org/issue14203 #14200: Idle shell crash on printing non-BMP unicode character http://bugs.python.org/issue14200 #14191: argparse: nargs='*' doesn't get out-of-order positional parame http://bugs.python.org/issue14191 #14183: Test coverage for packaging.install and packaging.pypi.wrapper http://bugs.python.org/issue14183 #14180: Factorize code to convert int/float to time_t, timeval or time http://bugs.python.org/issue14180 #14179: Test coverage for lib/re.py http://bugs.python.org/issue14179 Top 10 most discussed issues (10) ================================= #7652: Merge C version of decimal into py3k. http://bugs.python.org/issue7652 23 msgs #14218: include rendered output in addition to markup http://bugs.python.org/issue14218 14 msgs #14127: os.stat and os.utime: allow preserving exact metadata http://bugs.python.org/issue14127 12 msgs #13719: bdist_msi upload fails http://bugs.python.org/issue13719 11 msgs #14191: argparse: nargs='*' doesn't get out-of-order positional parame http://bugs.python.org/issue14191 11 msgs #14228: It is impossible to catch sigint on startup in python code http://bugs.python.org/issue14228 10 msgs #14216: ImportError: No module named binascii http://bugs.python.org/issue14216 8 msgs #11379: Remove "lightweight" from minidom description http://bugs.python.org/issue11379 7 msgs #13964: os.utimensat() and os.futimes() should accept (sec, nsec), dro http://bugs.python.org/issue13964 7 msgs #14200: Idle shell crash on printing non-BMP unicode character http://bugs.python.org/issue14200 7 msgs Issues closed (43) ================== #5689: Support xz compression in tarfile module http://bugs.python.org/issue5689 closed by nadeem.vawda #6715: xz compressor support http://bugs.python.org/issue6715 closed by nadeem.vawda #10369: tarfile requires an actual file on disc; a file-like object is http://bugs.python.org/issue10369 closed by lars.gustaebel #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 closed by pitrou #12568: Add functions to get the width in columns of a character http://bugs.python.org/issue12568 closed by loewis #13550: Rewrite logging hack of the threading module http://bugs.python.org/issue13550 closed by python-dev #13704: Random number generator in Python core http://bugs.python.org/issue13704 closed by christian.heimes #13860: PyBuffer_FillInfo() return value http://bugs.python.org/issue13860 closed by skrah #13882: PEP 410: Use decimal.Decimal type for timestamps http://bugs.python.org/issue13882 closed by haypo #13970: frameobject should not have f_yieldfrom attribute http://bugs.python.org/issue13970 closed by python-dev #13981: time.sleep() should use nanosleep() if available http://bugs.python.org/issue13981 closed by haypo #14071: allow more than one hash seed per process (move _Py_HashSecret http://bugs.python.org/issue14071 closed by haypo #14085: PyUnicode_WRITE: "comparison is always true" warnings http://bugs.python.org/issue14085 closed by python-dev #14100: Add a missing info to PEP 393 + link from whatsnew 3.3 http://bugs.python.org/issue14100 closed by loewis #14114: 2.7.3rc1 chm gives JS error http://bugs.python.org/issue14114 closed by loewis #14122: operator: div() instead of truediv() in documention since 3.1. http://bugs.python.org/issue14122 closed by ezio.melotti #14144: urllib2 HTTPRedirectHandler not forwarding POST data in redire http://bugs.python.org/issue14144 closed by orsenthil #14166: private dispatch table for picklers http://bugs.python.org/issue14166 closed by pitrou #14168: Bug in minidom 3.3 after optimization patch http://bugs.python.org/issue14168 closed by loewis #14171: warnings from valgrind about openssl as used by CPython http://bugs.python.org/issue14171 closed by loewis #14172: ref-counting leak in buffer usage in Python/marshal.c http://bugs.python.org/issue14172 closed by pitrou #14176: Fix unicode literals http://bugs.python.org/issue14176 closed by terry.reedy #14177: marshal.loads accepts unicode strings http://bugs.python.org/issue14177 closed by pitrou #14178: _elementtree problem deleting slices with steps != +1 http://bugs.python.org/issue14178 closed by eli.bendersky #14181: Support getbuffer redirection scheme in memoryview http://bugs.python.org/issue14181 closed by skrah #14188: Sharing site-packages between Apple and python.org builds brea http://bugs.python.org/issue14188 closed by ned.deily #14192: stdout.encoding not set when redirecting windows command line http://bugs.python.org/issue14192 closed by loewis #14193: broken link on PEP 385 http://bugs.python.org/issue14193 closed by georg.brandl #14194: typo in pep414 http://bugs.python.org/issue14194 closed by georg.brandl #14195: WeakSet has ordering relations wrong http://bugs.python.org/issue14195 closed by meador.inge #14199: Keep a refence to mro in _PyType_Lookup() and super_getattro() http://bugs.python.org/issue14199 closed by python-dev #14205: Raise an error if a dict is modified during a lookup http://bugs.python.org/issue14205 closed by python-dev #14211: Don't rely on borrowed _PyType_Lookup() reference in PyObject_ http://bugs.python.org/issue14211 closed by haypo #14212: Segfault when using re.finditer over mmap http://bugs.python.org/issue14212 closed by python-dev #14213: python33.dll not removed on uninstallation http://bugs.python.org/issue14213 closed by vinay.sajip #14217: text output pretends to be code http://bugs.python.org/issue14217 closed by orsenthil #14220: "yield from" kills generator on re-entry http://bugs.python.org/issue14220 closed by python-dev #14221: re.sub backreferences to numbered groups produce garbage http://bugs.python.org/issue14221 closed by ezio.melotti #14223: curses addch broken on Python3.3a1 http://bugs.python.org/issue14223 closed by python-dev #14231: Fix or drop Lib/test/crashers/borrowed_ref_1.py http://bugs.python.org/issue14231 closed by haypo #14233: argparse: "append" action fails to override default values http://bugs.python.org/issue14233 closed by r.david.murray #14240: lstrip problem http://bugs.python.org/issue14240 closed by amaury.forgeotdarc #1469629: __dict__ = self in subclass of dict causes a memory leak http://bugs.python.org/issue1469629 closed by python-dev

1 0

Adding a builtins parameter to eval(), exec() and __import__().
by Mark Shannon 09 Mar '12

09 Mar '12

I propose adding an optional (keyword-only?) 3rd parameter, "builtins" to exec(), eval(), __import__() and any other functions that take locals and globals as parameters. Currently, Python code is evaluated in a context of three name spaces; locals(), globals() and builtins. However, eval & exec only take 2 (optional) namespaces as parameters; locals and globals, so access to builtins is poorly defined. The reason I am proposing this here rather than on python-ideas is that treating the triple of [locals, globals, builtins] as a single "execution context" can be implemented in a really nice way. Internally, the execution context of [locals, globals, builtins] can be treated a single immutable object (custom object or tuple) Treating it as immutable means that it can be copied merely by taking a reference. A nice trick in the implementation is to make a NULL locals mean "fast" locals for function contexts. Frames, could then acquire their globals and builtins by a single reference copy from the function object, rather than searching globals for a '__builtins__' to find the builtins. A unified execution context will speed up all calls (to Python functions) as frame allocation and deallocation would be faster. I used this implementation in my original HotPy VM, and it worked well. It should also help with sandboxing, as it would make it easier to analyse and thus control access to builtins, since the execution context of all code would be easier to determine. Currently, it is impossible to allow one function access to sensitive functions like open(), while denying it to others, as any code can then get the builtins of another function via f.__globals__['builtins__']. Separating builtins from globals could solve this. Cheers, Mark.

9 18

FYI dict lookup now raises a RuntimeError if the dict is modified during the lookup
by Victor Stinner 09 Mar '12

09 Mar '12

If you use your own comparaison function (__eq__) for objects used as dict keys, you may now get RuntimeError with Python 3.3 if the dict is modified during a dict lookup in a multithreaded application. You should use a lock on the dict to avoid this exception. Said differently, a dict lookup is not atomic if you use your own types as dict keys and these types implement __eq__. In Python < 3.3, the dict lookup is retried until the dict is no more modified during the lookup, which leads to a stack overflow crash if the dict is always modified. See the issue #14205 for the rationale. Victor

1 0

PEP 414 - some numbers from the Django port
by Vinay Sajip 09 Mar '12

09 Mar '12

PEP 414 mentions the use of function wrappers and talks about both their obtrusiveness and performance impact on Python code. In the Django Python 3 port, I've used unicode_literals, and hence have no u prefixes in the ported code, and use a function wrapper to adorn native strings where they are needed. Though the port is still work in progress, it passes all tests on 2.x and 3.x with the SQLite adapter, with only a small number skipped specifically during the porting exercise (generally due to representational differences). I'd like to share some numbers from this port to see what people here think about them. Firstly, on obtrusiveness: Out of a total of 1872 source files, the native string marker only appears in 30 files - 18 files in Django itself, and 12 files in the test suite. This is less than 2% of files, so the native string markers are not especially invasive when looking at code. There are only 76 lines in the ported Django which contain native string markers. Secondly, on performance. I ran the following steps 6 times: Run the test suite on unported Django using Python 2.7.2 ("vanilla") Run the test suite on the ported Django using Python 2.7.2 ("ported") Run the test suite on the ported Django using Python 3.2.2 ("ported3") Django skips some tests because dependencies aren't installed (e.g. PIL for Python 3.2). The raw numbers, in seconds elapsed for the test run, are given below: vanilla (4659 tests): 468.586 486.231 467.584 464.916 480.530 475.457 ported (4655 tests): 467.350 480.902 479.276 478.748 478.115 486.044 ported3 (4609 tests): 463.161 470.423 463.833 448.097 456.727 504.402 If we allow for the different numbers of tests run by dividing by the number of tests and multiplying by 100, we get: vanilla-weighted: 10.057 10.436 10.036 9.979 10.314 10.205 ported-weighted: 10.040 10.331 10.296 10.285 10.271 10.441 ported3-weighted: 10.049 10.207 10.064 9.722 9.909 10.944 If I run these through ministat, it tells me there is no significant difference in these data sets, with a 95% confidence level: $ ministat -w 74 vanilla-weighted ported-weighted ported3-weighted x vanilla-weighted + ported-weighted * ported3-weighted +--------------------------------------------------------------------------+ | * + | |* * x ** * ++x+ * *| ||_______________|___M____|AA_M___AM___|__|_________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 6 9.979 10.436 10.205 10.171167 0.17883782 + 6 10.04 10.441 10.296 10.277333 0.13148485 No difference proven at 95.0% confidence * 6 9.722 10.944 10.064 10.149167 0.42250274 No difference proven at 95.0% confidence So, looking at a large project in a relevant problem domain, unicode_literals and native string markers would appear not to adversely impact readability or performance. Your comments would be appreciated. Regards, Vinay Sajip

4 5

Sandboxing Python
by Victor Stinner 08 Mar '12

08 Mar '12

Hi, The frozendict discussion switched somewhere to sandboxing, and so I prefer to start a new thread. There are various ways to implement a sandbox, but I would like to expose here how I implemented pysandbox to have your opinion. pysandbox is written to execute quickly a short untrusted function in a sandbox and then continue the normal execution of the program. It is possible to "enable" the sandbox, but also later to "disable" it. It is written for Python using only one thread and one process. To create a sandbox, pysandbox uses various protections. The main idea is to create an empty namespace and ensure that it is not possible to use objects added into the namespace for escaping from the sandbox. pysandbox only uses one thread and one process and so it doesn't replace the existing trusted namespace, but create a new one. The security of pysandbox depends on the sandbox namespace sealing. I don't want to integrate pysandbox in CPython because I am not yet conviced that the approach is secure by design. I am trying to patch Python to help the implementation of Python security modules and of read-only proxies. You can find below the list of protections implemented in pysandbox. Some of them are implemented in C. I challenge anymore to break pysandbox! I would be happy if anyone breaks it because it would make it more stronger. https://github.com/haypo/pysandbox/ http://pypi.python.org/pypi/pysandbox Namespace ========= * Make builtins read only * Remove function attribute: * frame.f_locals * function.func_closure/__closure__ * function.func_defaults/__defaults__ * function.func_globals/__globals__ * type.__subclasses__ * builtin_function.__self__ * Workaround the lack of frozendict, remove dict attributes: *__init__ * clear * __delitem__ * pop * popitem * setdefault * __setitem__ * update * Create a proxy for objects injected to the sandbox namespace and for the result of functions (the result of callable objects is also proxified) Generic ======= Remove all builtin symbols not in the whitelist. Features ======== import ------ * Replace __import__ function to use an import whitelist Filesystem ---------- * Replace open and file functions to deny access to the filesystem Exit ---- * Replace exit function * Remove SystemExit builtin exception Standard input/output --------------------- * Replace sys.stdin, sys.stdout and sys.stderr Bytecode ======== Execute arbitrary bytecode may crash Python, or lead to execution of arbitrary (machine) code. * Patch code.__new__() * Remove attributes: * function.func_code/__code__ * frame.f_code * generator.gi_code Victor

11 33