Mailman 3 October 2011 - Python-Dev

Re: [Python-Dev] memcmp performance
by Richard Saunders Oct. 24, 2011

Oct. 24, 2011

>>> Richard Saunders >>> I have been doing some performance experiments with memcmp, and I was >>> surprised that memcmp wasn't faster than it was in Python. I did a whole, >>> long analysis and came up with some very simple results. >> >>Antoine Pitrou, 20.10.2011 23:08: >> Thanks for the analysis. Non-bugfix work now happens on Python 3, where >> the str type is Python 2's unicode type. Your recommendations would >> have to be revisited under that light. > > Stefan Behnel <stefan_ml(a)behnel.de> >Well, Py3 is quite a bit different now that PEP393 is in. It appears to use >memcmp() or strcmp() a lot less than before, but I think unicode_compare() >should actually receive an optimisation to use a fast memcmp() if both >string kinds are equal, at least when their character unit size is less >than 4 (i.e. especially for ASCII strings). Funny enough, tailmatch() has >such an optimisation. I started looking at the most recent 3.x baseline: a lot of places, the memcmp analysis appears relevant (zlib, arraymodule, datetime, xmlparse): all still use memcmp in about the same way. But I agree that there are some major differences in the unicode portion. As long as the two strings are the same unicode "kind", you can use a memcmp to compare. In that case, I would almost argue some memcmp optimization is even more important: unicode strings are potentially 2 to 4 times larger, so the amount of time spent in memcmp may be more (i.e., I am still rooting for -fno-builtin-memcmp on the compile lines). I went ahead a quick string_test3.py for comparing strings (similar to what I did in Python 2.7) # Simple python string comparison test for Python 3.3 a = []; b = []; c = []; d = [] for x in range(0,1000) : a.append("the quick brown fox"+str(x)) b.append("the wuick brown fox"+str(x)) c.append("the quick brown fox"+str(x)) d.append("the wuick brown fox"+str(x)) count = 0 for x in range(0,200000) : if a==c : count += 1 if a==c : count += 2 if a==d : count += 3 if b==c : count += 5 if b==d : count += 7 if c==d : count += 11 print(count) Timings on On My FC14 machine (Intel Xeon W3520(a)2.67Ghz): 29.18 seconds: Vanilla build of Python 3.3 29.17 seconds: Python 3.3 compiled with -fno-builtin-memcmp: No change: a little investigation shows unicode_compare is where all the work is: Here's currently the main loop inside unicode_compare: for (i = 0; i < len1 && i < len2; ++i) { Py_UCS4 c1, c2; c1 = PyUnicode_READ(kind1, data1, i); c2 = PyUnicode_READ(kind2, data2, i); if (c1 != c2) return (c1 < c2) ? -1 : 1; } return (len1 < len2) ? -1 : (len1 != len2); If both loops are the same unicode kind, we can add memcmp to unicode_compare for an optimization: Py_ssize_t len = (len1<len2) ? len1: len2; /* use memcmp if both the same kind */ if (kind1==kind2) { int result=memcmp(data1, data2, ((int)kind1)*len); if (result!=0) return result<0 ? -1 : +1; } Rerunning the test with this small change to unicode_compare: 17.84 seconds: -fno-builtin-memcmp 36.25 seconds: STANDARD memcmp The standard memcmp is WORSE that the original unicode_compare code, but if we compile using memcmp with -fno-builtin-memcmp, we get that wonderful 2x performance increase again. I am still rooting for -fno-builtin-memcmp in both Python 2.7 and 3.3 ... (after we put memcmp in unicode_compare) Gooday, Richie

5 5

Re: [Python-Dev] cpython (2.7): Whoops, PyException_GetTraceback() is not documented on 2.7
by Georg Brandl Oct. 24, 2011

Oct. 24, 2011

On 10/23/11 20:54, petri.lehtinen wrote: > http://hg.python.org/cpython/rev/5c4781a237ef > changeset: 73073:5c4781a237ef > branch: 2.7 > parent: 73071:11da12600f5b > user: Petri Lehtinen <petri(a)digip.org> > date: Sun Oct 23 21:52:10 2011 +0300 > summary: > Whoops, PyException_GetTraceback() is not documented on 2.7 If it exists there, why not document it instead? :) Georg

2 2

Re: [Python-Dev] PEP397 no command line options to python?
by Paul Moore Oct. 24, 2011

Oct. 24, 2011

(Sorry, should have gone to the list...) On 22 October 2011 13:15, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > Nick Coghlan <ncoghlan <at> gmail.com> writes: > >> As a simpler alternative, I suggest the launcher just gain a "--which" >> long option that displays the full path to the interpreter it found. >> >> So: >> >> C:\> py -2 --which >> C:\Python27\python.exe >> >> C:\> py -3 --which >> C:\Python32\python.exe >> >> No significant complexity in the launcher, and if you want to add >> additional arguments like -m, -c, or -i you can do it by running >> '--which' and switching to invoking that interpreter directly. > > Perhaps even simpler would be for the -h option to print the interpreter paths > which would be returned for -2 and -3, on separate lines, even without the > --which, e.g. > > Currently configured: > -2: c:\Python27\python.exe > -3: c:\Python32\python.exe --which is nice for people who can use Unix-style $() or Powershell & to directly execute the output as a command. & (py -3 --which) Paul >

5 9

AUTO: Jon K Peck is out of the office (returning 10/27/2011)
by Jon K Peck Oct. 23, 2011

Oct. 23, 2011

I am out of the office until 10/27/2011. I am out of the office attending the IBM IOD conference. I will be back in the office on Friday, 10/28. My email responses will be delayed during that period. Note: This is an automated response to your message "Python-Dev Digest, Vol 99, Issue 45" sent on 10/23/11 4:00:03. This is the only notification you will receive while this person is away.

1 0

PEP397 no command line options to python?
by Sam Partington Oct. 22, 2011

Oct. 22, 2011

Hello all, I was surprised to see that the excellent pylauncher doesn't do the magic shebang processing if you give it any python command line options. e.g. Given #!/usr/bin/env python2.6 import sys print(sys.executable) C:\>py test.py C:\Python26\python.exe C:\>py -utt test.py C:\Python27\python.exe It is spelled out that it shouldn't do so in the pep : "Only the first command-line argument will be checked for a shebang line and only if that argument does not start with a '-'." But I can't really see why that should be the case. What is the rational behind this? It is very surprising to the user that adding a simple option like -tt should change the way the launcher behaves. The PEP also states that the launcher should show the python help if '-h' is specified : "If the only command-line argument is "-h" or "--help", the launcher will print a small banner and command-line usage, then pass the argument to the default Python. This will cause help for the launcher being printed followed by help for Python itself. The output from the launcher will clearly indicate the extended help information is coming from the launcher and not Python." To me that would suggest to end users that they can use any of the command line options with the launcher, and they should behave as if you had called python directly. I am directing this to python-dev because this pylauncher is merely implementing the PEP as it currently stands, so the PEP would ideally need to be modified before the launcher. I would change the that first paragraph of the PEP to read something like : " The first command-line argument not beginning with a '-' will be checked for a shebang line, but only if : * That command-line argument is not preceded by a '-c' or '-m', and providing * There is no explicit version specifier, e.g. -2.7 as documumented later in this PEP. " Incidentally whilst implementing this I also noticed a bug in the pylauncher whereby the py launcher would incorrectly treat "py t3" as a request for python version as if '-3' had been specified. I have a small patch that fixes this and implements the above for pylauncher with extra tests if there is interest. Sam PS I have been lurking for a while, hello everyone.

6 17

[PATCH] unicode subtypes broken in latest py3k debug builds
by Stefan Behnel Oct. 22, 2011

Oct. 22, 2011

Hi, the py3k debug build has been broken in Cython's integration tests for a couple of weeks now due to a use-after-decref bug. Here's the fix, please apply. BTW, is there a reason unicode_subtype_new() copies the buffer of the unicode object it just created, instead of just stealing it? Stefan

2 1

Re: [Python-Dev] [Python-checkins] cpython: Document that packaging doesn’t create __init__.py files (#3902).
by Nick Coghlan Oct. 21, 2011

Oct. 21, 2011

On Fri, Oct 21, 2011 at 11:52 PM, eric.araujo <python-checkins(a)python.org> wrote: > +To distribute extension modules that live in a package (e.g. ``package.ext``), > +you need to create you need to create a :file:`{package}/__init__.py` file to > +let Python recognize and import your module. "you need to create" is repeated in the new text. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-Dev] [Python-checkins] cpython: Issue 13227: Option to make the lru_cache() type specific (suggested by Andrew
by Nick Coghlan Oct. 21, 2011

Oct. 21, 2011

On Fri, Oct 21, 2011 at 1:57 AM, raymond.hettinger <python-checkins(a)python.org> wrote: > + If *typed* is set to True, function arguments of different types will be > + cached separately. For example, ``f(3)`` and ``f(3.0)`` will be treated > + as distinct calls with distinct results. I've been pondering this one a bit since reviewing it on the tracker, and I'm wondering if we have the default behaviour the wrong way around. For "typed=True": - never results in accidental type coercion and potentially wrong answers* (see below) - cache uses additional memory (each entry is larger, more entries may be stored) - additional cache misses - differs from current behaviour For "typed=False" (current default): - matches current (pre-3.3) behaviour - can lead to accidental type coercion and incorrect answers I only just realised this morning that the existing (untyped) caching behaviour can give answers that are *numerically* wrong, not just of the wrong type. This becomes clear once we use division as our test operation rather than multiplication and bring Decimal into the mix: >>> from functools import lru_cache >>> @lru_cache() ... def divide(x, y): ... return x / y ... >>> from decimal import Decimal >>> 10 / 9 1.1111111111111112 >>> Decimal(10) / Decimal(9) Decimal('1.111111111111111111111111111') >>> divide(10, 9) 1.1111111111111112 >>> divide(Decimal(10), Decimal(9)) 1.1111111111111112 At the very least, I think lru_cache should default to typed behaviour in 3.3, with people being able to explicitly switch it off as a cache optimisation technique when they know it doesn't matter. You could even make the case that making the cache type aware under the hood in 3.2 would be a bug fix, and the only new feature in 3.3 would be the ability to switch off the type awareness to save memory. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Status of the PEP 400? (deprecate codecs.StreamReader/StreamWriter)
by Victor Stinner Oct. 21, 2011

Oct. 21, 2011

Hi, Three weeks ago, I posted a draft on my PEP on this mailing list. I tried to include all remarks you made, and the PEP is now online: http://www.python.org/dev/peps/pep-0400/ It's now unclear to me if the PEP will be accepted or rejected. I don't know what to do to move forward. Victor

6 11

Re: [Python-Dev] [Python-checkins] cpython (3.2): adjust braces a bit
by Benjamin Peterson Oct. 21, 2011

Oct. 21, 2011

2011/10/21 Eric V. Smith <eric(a)trueblade.com>: > On 10/21/2011 12:31 PM, Benjamin Peterson wrote: >> 2011/10/21 Eric V. Smith <eric(a)trueblade.com>: >>> What's the logic for adding some braces, but removing others? >> >> No braces if everything is a one-liner, otherwise braces everywhere. > > Not sure what "everything" means here. My specific question is why > braces were added here: > - if (tb == Py_None) > + if (tb == Py_None) { > tb = NULL; > + } Because an else if follows it. -- Regards, Benjamin

1 0