Mailman 3 September 2011 - Python-checkins

Hg tips (was Re: cpython (merge default -> default): Merge heads.)
by Ezio Melotti Oct. 2, 2011

Oct. 2, 2011

Tip 1 -- merging heads: A while ago Éric suggested a nice tip to make merges easier and since I haven't seen many people using it and now I got a chance to use it again, I think it might be worth showing it once more: # so assume you just committed some changes: $ hg ci Doc/whatsnew/3.3.rst -m 'Update and reorganize the whatsnew entry for PEP 393.' # you push them, but someone else pushed something in the meanwhile, so the push fails $ hg push pushing to ssh://hg@hg.python.org/cpython searching for changes abort: push creates new remote heads on branch 'default'! (you should pull and merge or use push -f to force) # so you pull the other changes $ hg pull -u pulling from ssh://hg@hg.python.org/cpython searching for changes adding changesets adding manifests adding file changes added 4 changesets with 5 changes to 5 files (+1 heads) not updating, since new heads added (run 'hg heads' to see heads, 'hg merge' to merge) # and use "hg heads ." to see the two heads (yours and the one you pulled) in the current branch $ hg heads . changeset: 72521:e6a2b54c1d16 tag: tip user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Thu Sep 29 04:02:13 2011 +0200 summary: Fix hex_digit_to_int() prototype: expect Py_UCS4, not Py_UNICODE changeset: 72517:ba6ee5cc9ed6 user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Thu Sep 29 08:34:36 2011 +0300 summary: Update and reorganize the whatsnew entry for PEP 393. # here comes the tip: before merging you switch to the other head (i.e. the one pushed by Victor), # if you don't switch, you'll be merging Victor changeset and in case of conflicts you will have to review # and modify his code (e.g. put a Misc/NEWS entry in the right section or something more complicated) $ hg up e6a2b54c1d16 6 files updated, 0 files merged, 0 files removed, 0 files unresolved # after the switch you will merge the changeset you just committed, so in case of conflicts # reviewing and merging is much easier because you know the changes already $ hg merge 1 files updated, 0 files merged, 0 files removed, 0 files unresolved (branch merge, don't forget to commit) # here everything went fine and there were no conflicts, and in the diff I can see my last changeset $ hg di diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst [...] # everything looks fine, so I can commit the merge and push $ hg ci -m 'Merge heads.' $ hg push pushing to ssh://hg@hg.python.org/cpython searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 2 changesets with 1 changes to 1 files remote: buildbot: 2 changes sent successfully remote: notified python-checkins(a)python.org of incoming changeset ba6ee5cc9ed6 remote: notified python-checkins(a)python.org of incoming changeset e7672fe3cd35 This tip is not only useful while merging, but it's also useful for python-checkins reviews, because the "merge" mail has the same diff of the previous mail rather than having 15 unrelated changesets from the last week because the committer didn't pull in a while. Tip 2 -- extended diffs: If you haven't already, enable git diffs, adding to your ~/.hgrc the following two lines: > [diff] > git = True > (this is already in the devguide, even if 'git = on' is used there. The mercurial website uses git = True too.) More info: http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/ Tip 3 -- extensions: I personally like the 'color' extension, it makes the output of commands like 'hg diff' and 'hg stat' more readable (e.g. it shows removed lines in red and added ones in green). If you want to give it a try, add to your ~/.hgrc the following two lines: > [extensions] > color = > If you find operations like pulling, updating or cloning too slow, you might also want to look at the 'progress' extension, which displays a progress bar during these operations: > [extensions] > progress = > Tip 4 -- porting from 2.7 to 3.2: The devguide suggests: > > hg export a7df1a869e4a | hg import --no-commit - > but it's not always necessary to copy the changeset number manually. If you are porting your last commit you can just use 'hg export 2.7' (or any other branch name): * using the one-dir-per-branch setup: wolf@hp:~/dev/py/2.7$ hg ci -m 'Fix some bug.' wolf@hp:~/dev/py/2.7$ cd ../3.2 wolf@hp:~/dev/py/3.2$ hg pull -u ../2.7 wolf@hp:~/dev/py/3.2$ hg export 2.7 | hg import --no-commit - * using the single-dir setup: wolf@hp:~/dev/python$ hg branch 2.7 wolf@hp:~/dev/python$ hg ci -m 'Fix some bug.' wolf@hp:~/dev/python$ hg up 3.2 # here you might enjoy the progress extension wolf@hp:~/dev/python$ hg export 2.7 | hg import --no-commit - And then you can check that everything is fine, and commit on 3.2 too. Of course it works the other way around (from 3.2 to 2.7) too. I hope you'll find these tips useful. Best Regards, Ezio Melotti On Thu, Sep 29, 2011 at 8:36 AM, ezio.melotti <python-checkins(a)python.org>wrote: > http://hg.python.org/cpython/rev/e7672fe3cd35 > changeset: 72522:e7672fe3cd35 > parent: 72520:e6a2b54c1d16 > parent: 72521:ba6ee5cc9ed6 > user: Ezio Melotti <ezio.melotti(a)gmail.com> > date: Thu Sep 29 08:36:23 2011 +0300 > summary: > Merge heads. > > files: > Doc/whatsnew/3.3.rst | 63 +++++++++++++++++++++---------- > 1 files changed, 42 insertions(+), 21 deletions(-) > >

2 1

peps: Update posted by Greg Ewing to python-ideas today
by guido.van.rossum Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/peps/rev/31aa6576828d changeset: 3953:31aa6576828d user: Guido van Rossum <guido(a)google.com> date: Fri Sep 30 16:08:30 2011 -0700 summary: Update posted by Greg Ewing to python-ideas today files: pep-0335.txt | 329 +++++++++++++++++++++++++++++++++++++- 1 files changed, 317 insertions(+), 12 deletions(-) diff --git a/pep-0335.txt b/pep-0335.txt --- a/pep-0335.txt +++ b/pep-0335.txt @@ -2,13 +2,13 @@ Title: Overloadable Boolean Operators Version: $Revision$ Last-Modified: $Date$ -Author: Greg Ewing <greg.ewing(a)canterbury.ac.nz> +Author: Gregory Ewing <greg(a)cosc.canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Aug-2004 -Python-Version: 2.4 -Post-History: 05-Sep-2004 +Python-Version: 3.3 +Post-History: 05-Sep-2004, 30-Sep-2011 Abstract @@ -45,7 +45,7 @@ operators excluded from those able to be customised can be inconvenient. Examples include: -1. Numeric/Numarray, in which almost all the operators are defined on +1. NumPy, in which almost all the operators are defined on arrays so as to perform the appropriate operation between corresponding elements, and return an array of the results. For consistency, one would expect a boolean operation between two @@ -54,7 +54,7 @@ There is a precedent for an extension of this kind: comparison operators were originally restricted to returning boolean results, - and rich comparisons were added so that comparisons of Numeric + and rich comparisons were added so that comparisons of NumPy arrays could return arrays of booleans. 2. A symbolic algebra system, in which a Python expression is @@ -87,7 +87,7 @@ 2. There must not be any appreciable loss of speed in the default case. -3. If possible, the customisation mechanism should allow the object to +3. Ideally, the customisation mechanism should allow the object to provide either short-circuiting or non-short-circuiting semantics, at its discretion. @@ -131,14 +131,14 @@ To permit short-circuiting, processing of the 'and' and 'or' operators is split into two phases. Phase 1 occurs after evaluation of the first operand but before the second. If the first operand defines the -appropriate phase 1 method, it is called with the first operand as +relevant phase 1 method, it is called with the first operand as argument. If that method can determine the result without needing the second operand, it returns the result, and further processing is skipped. If the phase 1 method determines that the second operand is needed, it returns the special value NeedOtherOperand. This triggers the -evaluation of the second operand, and the calling of an appropriate +evaluation of the second operand, and the calling of a relevant phase 2 method. During phase 2, the __and2__/__rand2__ and __or2__/__ror2__ method pairs work as for other binary operators. @@ -149,7 +149,7 @@ no corresponding phase 1 method, the second operand is always evaluated and the phase 2 method called. This allows an object which does not want short-circuiting semantics to simply implement the -relevant phase 2 methods and ignore phase 1. +phase 2 methods and ignore phase 1. Bytecodes @@ -182,9 +182,9 @@ Type Slots ---------- -A the C level, the new special methods are manifested as five new +At the C level, the new special methods are manifested as five new slots in the type object. In the patch, they are added to the -tp_as_number substructure, since this allowed making use of some +tp_as_number substructure, since this allows making use of some existing code for dealing with unary and binary operators. Their existence is signalled by a new type flag, Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD. @@ -208,7 +208,312 @@ PyObject *PyObject_LogicalAnd1(PyObject *); PyObject *PyObject_LogicalOr1(PyObject *); PyObject *PyObject_LogicalAnd2(PyObject *, PyObject *); - PyObject *PyObject_LogicalOr2(PyObject *, PyObject *); + + + +Alternatives and Optimisations +============================== + +This section discusses some possible variations on the proposal, +and ways in which the bytecode sequences generated for boolean +expressions could be optimised. + +Reduced special method set +-------------------------- + +For completeness, the full version of this proposal includes a +mechanism for types to define their own customised short-circuiting +behaviour. However, the full mechanism is not needed to address the +main use cases put forward here, and it would be possible to +define a simplified version that only includes the phase 2 +methods. There would then only be 5 new special methods (__and2__, +__rand2__, __or2__, __ror2__, __not__) with 3 associated type slots +and 3 API functions. + +This simplified version could be expanded to the full version +later if desired. + +Additional bytecodes +-------------------- + +As defined here, the bytecode sequence for code that branches on +the result of a boolean expression would be slightly longer than +it currently is. For example, in Python 2.7, + +:: + + if a and b: + statement1 + else: + statement2 + +generates + +:: + + LOAD_GLOBAL a + POP_JUMP_IF_FALSE false_branch + LOAD_GLOBAL b + POP_JUMP_IF_FALSE false_branch + <code for statement1> + JUMP_FORWARD end_branch + false_branch: + <code for statement2> + end_branch: + +Under this proposal as described so far, it would become something like + +:: + + LOAD_GLOBAL a + LOGICAL_AND_1 test + LOAD_GLOBAL b + LOGICAL_AND_2 + test: + POP_JUMP_IF_FALSE false_branch + <code for statement1> + JUMP_FORWARD end_branch + false_branch: + <code for statement2> + end_branch: + +This involves executing one extra bytecode in the short-circuiting +case and two extra bytecodes in the non-short-circuiting case. + +However, by introducing extra bytecodes that combine the logical +operations with testing and branching on the result, it can be +reduced to the same number of bytecodes as the original: + +:: + + LOAD_GLOBAL a + AND1_JUMP true_branch, false_branch + LOAD_GLOBAL b + AND2_JUMP_IF_FALSE false_branch + true_branch: + <code for statement1> + JUMP_FORWARD end_branch + false_branch: + <code for statement2> + end_branch: + +Here, AND1_JUMP performs phase 1 processing as above, +and then examines the result. If there is a result, it is popped +from the stack, its truth value is tested and a branch taken to +one of two locations. + +Otherwise, the first operand is left on the stack and execution +continues to the next bytecode. The AND2_JUMP_IF_FALSE bytecode +performs phase 2 processing, pops the result and branches if +it tests false + +For the 'or' operator, there would be corresponding OR1_JUMP +and OR2_JUMP_IF_TRUE bytecodes. + +If the simplified version without phase 1 methods is used, then +early exiting can only occur if the first operand is false for +'and' and true for 'or'. Consequently, the two-target AND1_JUMP and +OR1_JUMP bytecodes can be replaced with AND1_JUMP_IF_FALSE and +OR1_JUMP_IF_TRUE, these being ordinary branch instructions with +only one target. + +Optimisation of 'not' +--------------------- + +Recent versions of Python implement a simple optimisation in +which branching on a negated boolean expression is implemented +by reversing the sense of the branch, saving a UNARY_NOT opcode. + +Taking a strict view, this optimisation should no longer be +performed, because the 'not' operator may be overridden to produce +quite different results from usual. However, in typical use cases, +it is not envisaged that expressions involving customised boolean +operations will be used for branching -- it is much more likely +that the result will be used in some other way. + +Therefore, it would probably do little harm to specify that the +compiler is allowed to use the laws of boolean algebra to +simplify any expression that appears directly in a boolean +context. If this is inconvenient, the result can always be assigned +to a temporary name first. + +This would allow the existing 'not' optimisation to remain, and +would permit future extensions of it such as using De Morgan's laws +to extend it deeper into the expression. + + +Usage Examples +============== + +Example 1: NumPy Arrays +----------------------- + +:: + + #----------------------------------------------------------------- + # + # This example creates a subclass of numpy array to which + # 'and', 'or' and 'not' can be applied, producing an array + # of booleans. + # + #----------------------------------------------------------------- + + from numpy import array, ndarray + + class BArray(ndarray): + + def __str__(self): + return "barray(%s)" % ndarray.__str__(self) + + def __and2__(self, other): + return (self & other) + + def __or2__(self, other): + return (self & other) + + def __not__(self): + return (self == 0) + + def barray(*args, **kwds): + return array(*args, **kwds).view(type = BArray) + + a0 = barray([0, 1, 2, 4]) + a1 = barray([1, 2, 3, 4]) + a2 = barray([5, 6, 3, 4]) + a3 = barray([5, 1, 2, 4]) + + print "a0:", a0 + print "a1:", a1 + print "a2:", a2 + print "a3:", a3 + print "not a0:", not a0 + print "a0 == a1 and a2 == a3:", a0 == a1 and a2 == a3 + print "a0 == a1 or a2 == a3:", a0 == a1 or a2 == a3 + +Example 1 Output +---------------- + +:: + + a0: barray([0 1 2 4]) + a1: barray([1 2 3 4]) + a2: barray([5 6 3 4]) + a3: barray([5 1 2 4]) + not a0: barray([ True False False False]) + a0 == a1 and a2 == a3: barray([False False False True]) + a0 == a1 or a2 == a3: barray([False False False True]) + + +Example 2: Database Queries +--------------------------- + +:: + + #----------------------------------------------------------------- + # + # This example demonstrates the creation of a DSL for database + # queries allowing 'and' and 'or' operators to be used to + # formulate the query. + # + #----------------------------------------------------------------- + + class SQLNode(object): + + def __and2__(self, other): + return SQLBinop("and", self, other) + + def __rand2__(self, other): + return SQLBinop("and", other, self) + + def __eq__(self, other): + return SQLBinop("=", self, other) + + + class Table(SQLNode): + + def __init__(self, name): + self.__tablename__ = name + + def __getattr__(self, name): + return SQLAttr(self, name) + + def __sql__(self): + return self.__tablename__ + + + class SQLBinop(SQLNode): + + def __init__(self, op, opnd1, opnd2): + self.op = op.upper() + self.opnd1 = opnd1 + self.opnd2 = opnd2 + + def __sql__(self): + return "(%s %s %s)" % (sql(self.opnd1), self.op, sql(self.opnd2)) + + + class SQLAttr(SQLNode): + + def __init__(self, table, name): + self.table = table + self.name = name + + def __sql__(self): + return "%s.%s" % (sql(self.table), self.name) + + + class SQLSelect(SQLNode): + + def __init__(self, targets): + self.targets = targets + self.where_clause = None + + def where(self, expr): + self.where_clause = expr + return self + + def __sql__(self): + result = "SELECT %s" % ", ".join([sql(target) for target in self.targets]) + if self.where_clause: + result = "%s WHERE %s" % (result, sql(self.where_clause)) + return result + + + def sql(expr): + if isinstance(expr, SQLNode): + return expr.__sql__() + elif isinstance(expr, str): + return "'%s'" % expr.replace("'", "''") + else: + return str(expr) + + + def select(*targets): + return SQLSelect(targets) + + +#-------------------------------------------------------------------------------- + + dishes = Table("dishes") + customers = Table("customers") + orders = Table("orders") + + query = select(customers.name, dishes.price, orders.amount).where( + customers.cust_id == orders.cust_id and orders.dish_id == dishes.dish_id + and dishes.name == "Spam, Eggs, Sausages and Spam") + + print repr(query) + print sql(query) + +Example 2 Output +---------------- + +:: + + <__main__.SQLSelect object at 0x1cc830> + SELECT customers.name, dishes.price, orders.amount WHERE + (((customers.cust_id = orders.cust_id) AND (orders.dish_id = + dishes.dish_id)) AND (dishes.name = 'Spam, Eggs, Sausages and Spam')) Copyright -- Repository URL: http://hg.python.org/peps

1 0

cpython: pyexat uses the new Unicode API
by victor.stinner Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/cpython/rev/a1be34457ccf changeset: 72548:a1be34457ccf user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Sat Oct 01 01:05:40 2011 +0200 summary: pyexat uses the new Unicode API files: Modules/pyexpat.c | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1234,11 +1234,13 @@ static PyObject * xmlparse_getattro(xmlparseobject *self, PyObject *nameobj) { - const Py_UNICODE *name; + Py_UCS4 first_char; int handlernum = -1; if (!PyUnicode_Check(nameobj)) goto generic; + if (PyUnicode_READY(nameobj)) + return NULL; handlernum = handlername2int(nameobj); @@ -1250,8 +1252,8 @@ return result; } - name = PyUnicode_AS_UNICODE(nameobj); - if (name[0] == 'E') { + first_char = PyUnicode_READ_CHAR(nameobj, 0); + if (first_char == 'E') { if (PyUnicode_CompareWithASCIIString(nameobj, "ErrorCode") == 0) return PyLong_FromLong((long) XML_GetErrorCode(self->itself)); @@ -1265,7 +1267,7 @@ return PyLong_FromLong((long) XML_GetErrorByteIndex(self->itself)); } - if (name[0] == 'C') { + if (first_char == 'C') { if (PyUnicode_CompareWithASCIIString(nameobj, "CurrentLineNumber") == 0) return PyLong_FromLong((long) XML_GetCurrentLineNumber(self->itself)); @@ -1276,7 +1278,7 @@ return PyLong_FromLong((long) XML_GetCurrentByteIndex(self->itself)); } - if (name[0] == 'b') { + if (first_char == 'b') { if (PyUnicode_CompareWithASCIIString(nameobj, "buffer_size") == 0) return PyLong_FromLong((long) self->buffer_size); if (PyUnicode_CompareWithASCIIString(nameobj, "buffer_text") == 0) -- Repository URL: http://hg.python.org/cpython

1 0

peps: Error in PEP example. Closes #13079.
by eric.smith Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/peps/rev/9a9bd05b9fca changeset: 3952:9a9bd05b9fca user: Eric V. Smith <eric(a)trueblade.com> date: Fri Sep 30 16:35:03 2011 -0400 summary: Error in PEP example. Closes #13079. files: pep-3101.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-3101.txt b/pep-3101.txt --- a/pep-3101.txt +++ b/pep-3101.txt @@ -339,7 +339,7 @@ whose format specifiers might look something like the arguments to the strftime() function: - "Today is: {0:a b d H:M:S Y}".format(datetime.now()) + "Today is: {0:%a %b %d %H:%M:%S %Y}".format(datetime.now()) For all built-in types, an empty format specification will produce the equivalent of str(value). It is recommended that objects -- Repository URL: http://hg.python.org/peps

1 0

cpython: Fix ResourceWarnings in makeunicodedata.py.
by ezio.melotti Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/cpython/rev/b0ef6a5a6dcc changeset: 72547:b0ef6a5a6dcc user: Ezio Melotti <ezio.melotti(a)gmail.com> date: Fri Sep 30 08:46:25 2011 +0300 summary: Fix ResourceWarnings in makeunicodedata.py. files: Tools/unicode/makeunicodedata.py | 167 +++++++++--------- 1 files changed, 87 insertions(+), 80 deletions(-) diff --git a/Tools/unicode/makeunicodedata.py b/Tools/unicode/makeunicodedata.py --- a/Tools/unicode/makeunicodedata.py +++ b/Tools/unicode/makeunicodedata.py @@ -816,15 +816,15 @@ expand=1, cjk_check=True): self.changed = [] - file = open_data(UNICODE_DATA, version) table = [None] * 0x110000 - while 1: - s = file.readline() - if not s: - break - s = s.strip().split(";") - char = int(s[0], 16) - table[char] = s + with open_data(UNICODE_DATA, version) as file: + while 1: + s = file.readline() + if not s: + break + s = s.strip().split(";") + char = int(s[0], 16) + table[char] = s cjk_ranges_found = [] @@ -855,32 +855,34 @@ self.table = table self.chars = list(range(0x110000)) # unicode 3.2 - file = open_data(COMPOSITION_EXCLUSIONS, version) self.exclusions = {} - for s in file: - s = s.strip() - if not s: - continue - if s[0] == '#': - continue - char = int(s.split()[0],16) - self.exclusions[char] = 1 + with open_data(COMPOSITION_EXCLUSIONS, version) as file: + for s in file: + s = s.strip() + if not s: + continue + if s[0] == '#': + continue + char = int(s.split()[0],16) + self.exclusions[char] = 1 widths = [None] * 0x110000 - for s in open_data(EASTASIAN_WIDTH, version): - s = s.strip() - if not s: - continue - if s[0] == '#': - continue - s = s.split()[0].split(';') - if '..' in s[0]: - first, last = [int(c, 16) for c in s[0].split('..')] - chars = list(range(first, last+1)) - else: - chars = [int(s[0], 16)] - for char in chars: - widths[char] = s[1] + with open_data(EASTASIAN_WIDTH, version) as file: + for s in file: + s = s.strip() + if not s: + continue + if s[0] == '#': + continue + s = s.split()[0].split(';') + if '..' in s[0]: + first, last = [int(c, 16) for c in s[0].split('..')] + chars = list(range(first, last+1)) + else: + chars = [int(s[0], 16)] + for char in chars: + widths[char] = s[1] + for i in range(0, 0x110000): if table[i] is not None: table[i].append(widths[i]) @@ -888,36 +890,39 @@ for i in range(0, 0x110000): if table[i] is not None: table[i].append(set()) - for s in open_data(DERIVED_CORE_PROPERTIES, version): - s = s.split('#', 1)[0].strip() - if not s: - continue - r, p = s.split(";") - r = r.strip() - p = p.strip() - if ".." in r: - first, last = [int(c, 16) for c in r.split('..')] - chars = list(range(first, last+1)) - else: - chars = [int(r, 16)] - for char in chars: - if table[char]: - # Some properties (e.g. Default_Ignorable_Code_Point) - # apply to unassigned code points; ignore them - table[char][-1].add(p) + with open_data(DERIVED_CORE_PROPERTIES, version) as file: + for s in file: + s = s.split('#', 1)[0].strip() + if not s: + continue - for s in open_data(LINE_BREAK, version): - s = s.partition('#')[0] - s = [i.strip() for i in s.split(';')] - if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS: - continue - if '..' not in s[0]: - first = last = int(s[0], 16) - else: - first, last = [int(c, 16) for c in s[0].split('..')] - for char in range(first, last+1): - table[char][-1].add('Line_Break') + r, p = s.split(";") + r = r.strip() + p = p.strip() + if ".." in r: + first, last = [int(c, 16) for c in r.split('..')] + chars = list(range(first, last+1)) + else: + chars = [int(r, 16)] + for char in chars: + if table[char]: + # Some properties (e.g. Default_Ignorable_Code_Point) + # apply to unassigned code points; ignore them + table[char][-1].add(p) + + with open_data(LINE_BREAK, version) as file: + for s in file: + s = s.partition('#')[0] + s = [i.strip() for i in s.split(';')] + if len(s) < 2 or s[1] not in MANDATORY_LINE_BREAKS: + continue + if '..' not in s[0]: + first = last = int(s[0], 16) + else: + first, last = [int(c, 16) for c in s[0].split('..')] + for char in range(first, last+1): + table[char][-1].add('Line_Break') # We only want the quickcheck properties # Format: NF?_QC; Y(es)/N(o)/M(aybe) @@ -928,31 +933,33 @@ # for older versions, and no delta records will be created. quickchecks = [0] * 0x110000 qc_order = 'NFD_QC NFKD_QC NFC_QC NFKC_QC'.split() - for s in open_data(DERIVEDNORMALIZATION_PROPS, version): - if '#' in s: - s = s[:s.index('#')] - s = [i.strip() for i in s.split(';')] - if len(s) < 2 or s[1] not in qc_order: - continue - quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No - quickcheck_shift = qc_order.index(s[1])*2 - quickcheck <<= quickcheck_shift - if '..' not in s[0]: - first = last = int(s[0], 16) - else: - first, last = [int(c, 16) for c in s[0].split('..')] - for char in range(first, last+1): - assert not (quickchecks[char]>>quickcheck_shift)&3 - quickchecks[char] |= quickcheck + with open_data(DERIVEDNORMALIZATION_PROPS, version) as file: + for s in file: + if '#' in s: + s = s[:s.index('#')] + s = [i.strip() for i in s.split(';')] + if len(s) < 2 or s[1] not in qc_order: + continue + quickcheck = 'MN'.index(s[2]) + 1 # Maybe or No + quickcheck_shift = qc_order.index(s[1])*2 + quickcheck <<= quickcheck_shift + if '..' not in s[0]: + first = last = int(s[0], 16) + else: + first, last = [int(c, 16) for c in s[0].split('..')] + for char in range(first, last+1): + assert not (quickchecks[char]>>quickcheck_shift)&3 + quickchecks[char] |= quickcheck for i in range(0, 0x110000): if table[i] is not None: table[i].append(quickchecks[i]) - zip = zipfile.ZipFile(open_data(UNIHAN, version)) - if version == '3.2.0': - data = zip.open('Unihan-3.2.0.txt').read() - else: - data = zip.open('Unihan_NumericValues.txt').read() + with open_data(UNIHAN, version) as file: + zip = zipfile.ZipFile(file) + if version == '3.2.0': + data = zip.open('Unihan-3.2.0.txt').read() + else: + data = zip.open('Unihan_NumericValues.txt').read() for line in data.decode("utf-8").splitlines(): if not line.startswith('U+'): continue -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (c75b04996f54): sum=0
by solipsis＠pitrou.net Sept. 30, 2011

Sept. 30, 2011

results for c75b04996f54 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogy4gYpQ', '-x']

1 0

cpython: Add PyUnicode_Copy() function, include it to the public API
by victor.stinner Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/cpython/rev/c75b04996f54 changeset: 72546:c75b04996f54 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Sep 30 02:26:44 2011 +0200 summary: Add PyUnicode_Copy() function, include it to the public API files: Include/unicodeobject.h | 5 +++ Modules/posixmodule.c | 3 +- Objects/unicodeobject.c | 42 +++++++++++++++------------- 3 files changed, 28 insertions(+), 22 deletions(-) diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -510,6 +510,11 @@ ); #endif +/* Get a copy of a Unicode string. */ +PyAPI_FUNC(PyObject*) PyUnicode_Copy( + PyObject *unicode + ); + /* Copy character from one unicode object into another, this function performs character conversion when necessary and falls back to memcpy if possible. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -729,8 +729,7 @@ else if (PyUnicode_Check(*param)) /* For a Unicode subtype that's not a Unicode object, return a true Unicode object with the same data. */ - *param = PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(*param), - PyUnicode_GET_SIZE(*param)); + *param = PyUnicode_Copy(*param); else *param = PyUnicode_FromEncodedObject(*param, Py_FileSystemDefaultEncoding, diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1209,6 +1209,20 @@ return NULL; } +PyObject* +PyUnicode_Copy(PyObject *unicode) +{ + if (!PyUnicode_Check(unicode)) { + PyErr_BadInternalCall(); + return NULL; + } + if (PyUnicode_READY(unicode)) + return NULL; + return PyUnicode_FromKindAndData(PyUnicode_KIND(unicode), + PyUnicode_DATA(unicode), + PyUnicode_GET_LENGTH(unicode)); +} + /* Widen Unicode objects to larger buffers. Return NULL if the string is too wide already. */ @@ -9061,9 +9075,7 @@ Py_INCREF(self); return (PyObject *) self; } - return PyUnicode_FromKindAndData(PyUnicode_KIND(self), - PyUnicode_DATA(self), - PyUnicode_GET_LENGTH(self)); + return PyUnicode_Copy(self); error: if (srelease && sbuf) PyMem_FREE(sbuf); @@ -10477,7 +10489,8 @@ return NULL; kind = PyUnicode_KIND(self); data = PyUnicode_1BYTE_DATA(self); - return PyUnicode_FromKindAndData(kind, data + PyUnicode_KIND_SIZE(kind, start), + return PyUnicode_FromKindAndData(kind, + data + PyUnicode_KIND_SIZE(kind, start), end-start); } @@ -11267,8 +11280,7 @@ return self; } else /* Subtype -- return genuine unicode string with the same value. */ - return PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(self), - PyUnicode_GET_SIZE(self)); + return PyUnicode_Copy(self); } PyDoc_STRVAR(swapcase__doc__, @@ -11453,10 +11465,7 @@ return (PyObject*) self; } else - return PyUnicode_FromUnicode( - PyUnicode_AS_UNICODE(self), - PyUnicode_GET_SIZE(self) - ); + return PyUnicode_Copy(self); } fill = width - _PyUnicode_LENGTH(self); @@ -11652,16 +11661,9 @@ "S.__sizeof__() -> size of S in memory, in bytes"); static PyObject * -unicode_getnewargs(PyUnicodeObject *v) -{ - PyObject *copy; - unsigned char *data; - int kind; - if (PyUnicode_READY(v) == -1) - return NULL; - kind = PyUnicode_KIND(v); - data = PyUnicode_1BYTE_DATA(v); - copy = PyUnicode_FromKindAndData(kind, data, PyUnicode_GET_LENGTH(v)); +unicode_getnewargs(PyObject *v) +{ + PyObject *copy = PyUnicode_Copy(v); if (!copy) return NULL; return Py_BuildValue("(N)", copy); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: PyUnicode_CopyCharacters() uses exceptions instead of assertions
by victor.stinner Sept. 30, 2011

Sept. 30, 2011

http://hg.python.org/cpython/rev/5d49d11f912d changeset: 72545:5d49d11f912d user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Sep 30 02:26:10 2011 +0200 summary: PyUnicode_CopyCharacters() uses exceptions instead of assertions Call PyErr_BadInternalCall() if inputs are not unicode strings. files: Objects/unicodeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -626,8 +626,10 @@ unsigned int from_kind, to_kind; void *from_data, *to_data; - assert(PyUnicode_Check(from)); - assert(PyUnicode_Check(to)); + if (!PyUnicode_Check(from) || !PyUnicode_Check(to)) { + PyErr_BadInternalCall(); + return -1; + } if (PyUnicode_READY(from)) return -1; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix a compiler warning
by victor.stinner Sept. 29, 2011

Sept. 29, 2011

http://hg.python.org/cpython/rev/e88fc377f961 changeset: 72544:e88fc377f961 user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Sep 30 01:55:49 2011 +0200 summary: Fix a compiler warning files: Modules/arraymodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2538,7 +2538,7 @@ if (n > 0) { arrayobject *self = (arrayobject *)a; Py_UCS4 *item = (Py_UCS4 *)self->ob_item; - item = (char *)PyMem_Realloc(item, n * sizeof(Py_UCS4)); + item = (Py_UCS4 *)PyMem_Realloc(item, n * sizeof(Py_UCS4)); if (item == NULL) { PyErr_NoMemory(); Py_DECREF(a); -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix array.array('u') constructor
by victor.stinner Sept. 29, 2011

Sept. 29, 2011

http://hg.python.org/cpython/rev/8b4828bcd67a changeset: 72543:8b4828bcd67a user: Victor Stinner <victor.stinner(a)haypocalc.com> date: Fri Sep 30 01:54:04 2011 +0200 summary: Fix array.array('u') constructor files: Modules/arraymodule.c | 18 ++++++++++++------ 1 files changed, 12 insertions(+), 6 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2529,19 +2529,25 @@ Py_DECREF(v); } else if (initial != NULL && PyUnicode_Check(initial)) { - Py_ssize_t n = PyUnicode_GET_DATA_SIZE(initial); + Py_ssize_t n; + if (PyUnicode_READY(initial)) { + Py_DECREF(a); + return NULL; + } + n = PyUnicode_GET_LENGTH(initial); if (n > 0) { arrayobject *self = (arrayobject *)a; - char *item = self->ob_item; - item = (char *)PyMem_Realloc(item, n); + Py_UCS4 *item = (Py_UCS4 *)self->ob_item; + item = (char *)PyMem_Realloc(item, n * sizeof(Py_UCS4)); if (item == NULL) { PyErr_NoMemory(); Py_DECREF(a); return NULL; } - self->ob_item = item; - Py_SIZE(self) = n / sizeof(Py_UCS4); - memcpy(item, PyUnicode_AS_DATA(initial), n); + self->ob_item = (char*)item; + Py_SIZE(self) = n; + if (!PyUnicode_AsUCS4(initial, item, n, 0)) + return NULL; self->allocated = Py_SIZE(self); } } -- Repository URL: http://hg.python.org/cpython

1 0