Mailman 3 January 2012 - Python-Dev

Proposed PEP on concurrent programming support
by Jim Jewett 04 Jan '12

04 Jan '12

(I've added back python-ideas, because I think that is still the appropriate forum.) >.... A new > suite type - the ``transaction`` will be added to the language. The > suite will have the semantics discussed above: modifying an object in > the suite will trigger creation of a thread-local shallow copy to be > used in the Transaction. Further modifications of the original will > cause all existing copies to be discarded and the transaction to be > restarted. ... How will you know that an object has been modified? The only ways I can think of are (1) Timestamp every object -- or at least every mutable object -- and hope that everybody agrees on which modifications should count. (2) Make two copies of every object you're using in the suite; at the end, compare one of them to both the original and the one you were operating on. With this solution, you can decide for youself what counts as a modification, but it still isn't straightforward; I would consider changing a value to be changing a dict, even though nothing in the item (header) itself changed. -jJ

1 0

PEP 7 clarification request: braces
by Nick Coghlan 04 Jan '12

04 Jan '12

I've been having an occasional argument with Benjamin regarding braces in 4-line if statements: if (cond) statement; else statement; vs. if (cond) { statement; } else { statement; } He keeps leaving them out, I occasionally tell him they should always be included (most recently this came up when we gave conflicting advice to a patch contributor). He says what he's doing is OK, because he doesn't consider the example in PEP 7 as explicitly disallowing it, I think it's a recipe for future maintenance hassles when someone adds a second statement to one of the clauses but doesn't add the braces. (The only time I consider it reasonable to leave out the braces is for one liner if statements, where there's no else clause at all) Since Benjamin doesn't accept the current brace example in PEP 7 as normative for the case above, I'm bringing it up here to seek clarification. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

23 43

Re: [Python-Dev] cpython: Add a new PyUnicode_Fill() function
by Antoine Pitrou 04 Jan '12

04 Jan '12

> +.. c:function:: int PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, \ > + Py_ssize_t length, Py_UCS4 fill_char) > + > + Fill a string with a character: write *fill_char* into > + ``unicode[start:start+length]``. > + > + Fail if *fill_char* is bigger than the string maximum character, or if the > + string has more than 1 reference. > + > + Return the number of written character, or return ``-1`` and raise an > + exception on error. The return type should then be Py_ssize_t, not int. Regards Antoine.

2 1

That depends on what the meaning of "is" is (was Re: http://mail.python.org/pipermail/python-dev/2011-December/115172.html)
by PJ Eby 03 Jan '12

03 Jan '12

On Sun, Jan 1, 2012 at 10:28 PM, Jim Jewett <jimjjewett(a)gmail.com> wrote: > Given the wording requiring a real dictionary, I would have assumed > that it was OK (if perhaps not sensible) to do pointer arithmetic and > access the keys/values/hashes directly. (Though if the breakage was > between python versions, I would feel guilty about griping too > loudly.) > If you're going to be a language lawyer about it, I would simply point out that all the spec requires is that "type(env) is dict" -- it says nothing about how Python defines "type" or "is" or "dict". So, you're on your own with that one. ;-)

2 3

Re: [Python-Dev] cpython: fix some possible refleaks from PyUnicode_READY error conditions
by Antoine Pitrou 02 Jan '12

02 Jan '12

On Mon, 02 Jan 2012 16:00:50 +0100 benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/d5cda62d0f8c > changeset: 74236:d5cda62d0f8c > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Jan 02 09:00:30 2012 -0600 > summary: > fix some possible refleaks from PyUnicode_READY error conditions > > files: > Objects/unicodeobject.c | 80 ++++++++++++++++++++-------- > 1 files changed, 56 insertions(+), 24 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -9132,10 +9132,15 @@ > Py_ssize_t len1, len2; > > str_obj = PyUnicode_FromObject(str); > - if (!str_obj || PyUnicode_READY(str_obj) == -1) > + if (!str_obj) > return -1; > sub_obj = PyUnicode_FromObject(substr); > - if (!sub_obj || PyUnicode_READY(sub_obj) == -1) { > + if (!sub_obj) { > + Py_DECREF(str_obj); > + return -1; > + } > + if (PyUnicode_READY(substr) == -1 || PyUnicode_READY(str_obj) == -1) { Shouldn't the first one be PyUnicode_READY(sub_obj) ?

2 1

[Python-Dev] http://mail.python.org/pipermail/python-dev/2011-December/115172.html
by Jim Jewett 02 Jan '12

02 Jan '12

In http://mail.python.org/pipermail/python-dev/2011-December/115172.html, P. J. Eby wrote: > On Sat, Dec 31, 2011 at 7:03 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote: >> While the dictionary probe has to start with a hash for backward >> compatibility reasons, is there a reason the overflow strategy for >> insertion has to be buckets containing lists? How about >> double-hashing, etc? > This won't help, because the keys still have the same hash value. ANYTHING > you do to them after they're generated will result in them still colliding. > The *only* thing that works is to change the hash function in such a way > that the strings end up with different hashes in the first place. > Otherwise, you'll still end up with (deliberate) collisions. Well, there is nothing wrong with switching to a different hash function after N collisions, rather than "in the first place". The perturbation effectively does by shoving the high-order bits through the part of the hash that survives the mask. > (Well, technically, you could use trees or some other O log n data > structure as a fallback once you have too many collisions, for some value > of "too many". Seems a bit wasteful for the purpose, though.) Your WSGI specification < http://www.python.org/dev/peps/pep-0333/ > requires using a real dictionary for compatibility; storing some of the values outside the values array would violate that. Do you consider that obsolete? -jJ

4 5

Hash collision security issue (now public)
by Jim Jewett 02 Jan '12

02 Jan '12

Victor Stinner wrote in <http://mail.python.org/pipermail/python-dev/2012-January/115198.html> > If we want to protect a website against this attack for example, we must > suppose that the attacker can inject arbitrary data and can get > (indirectly) the result of hash(str) (e.g. with the representation of a > dict in a traceback, with a JSON output, etc.). (1) Is it common to hash non-string input? Because generating integers that collide for certain dict sizes is pretty easy... (2) Would it make sense for traceback printing to sort dict keys? (Any site worried about this issue should already be hiding tracebacks from untrusted clients, but the cost of this extra protection may be pretty small, given that tracebacks shouldn't be printed all that often in the first place.) (3) Should the docs for json.encoder.JSONEncoder suggest sort_keys=True? -jJ

1 0

Hash collision security issue (now public)
by Jim Jewett 02 Jan '12

02 Jan '12

Paul McMillan in <http://mail.python.org/pipermail/python-dev/2012-January/115183.html> wrote: > Guido van Rossum wrote: >> Hm. I'm not sure I like the idea of extra arithmetic for every character >> being hashed. > the collision generator doesn't necessarily vary the length of the > string. Additionally, if we don't vary based on all the letters in the > string, an attacker can fix the characters that we do use and generate > colliding strings around them. If the new hash algorithm doesn't kick in before, say, 32 characters, then most currently hashed strings will not be affected. And if the attacker has to add 32 characters to every key, it reduces the "this can be done with only N bytes uploaded" risk. (The same logic would apply to even longer prefixes, except that an attacker might more easily find short-enough strings that collide.) > We could also consider a less computationally expensive operation > than the modulo for calculating the lookup index, like simply truncating > to the correct number of bits. Given that the modulo is always 2^N, how is that different? -jJ

1 0

Hash collision security issue (now public)
by Jim Jewett 01 Jan '12

01 Jan '12

Steven D'Aprano (in <http://mail.python.org/pipermail/python-dev/2011-December/115162.html>) wrote: > By compile-time, do you mean when the byte-code is compilated, i.e. just > before runtime, rather than a switch when compiling the Python executable from > source? No. I really mean when the C code is initially compiled to produce an python executable. The only reason we're worrying about this is that an adversary may force worst-case performance. If the python instance isn't a server, or at least isn't exposed to untrusted clients, then even a single extra "if" test is unjustified overhead. Adding overhead to every string hash or every dict lookup is bad. That said, adding some overhead (only) to dict lookups *that already hit half a dozen consecutive collisions* probably is reasonable, because that won't happen very often with normal data. (6 collisions can't happen at all unless there are already at least 6 entries, so small dicts are safe; with at least 1/3 of the slots empty, it should happen only 1/729 for worst-size larger dicts.) -jJ

1 0