Mailman 3 December 2008 - Python-Dev

Re: [Python-Dev] [Python-checkins] r67511 - in python/trunk: Doc/library/logging.rst Lib/logging/__init__.py Lib/test/test_logging.py Misc/NEWS
by Nick Coghlan 04 Dec '08

04 Dec '08

vinay.sajip wrote: > +def _showwarning(message, category, filename, lineno, file=None, line=None): > + """ > + Implementation of showwarnings which redirects to logging, which will first > + check to see if the file parameter is None. If a file is specified, it will > + delegate to the original warnings implementation of showwarning. Otherwise, > + it will call warnings.formatwarning and will log the resulting string to a > + warnings logger named "py.warnings" with level logging.WARNING. > + """ > + if file is not None: > + if _warnings_showwarning is not None: > + _warnings_showwarning(message, category, filename, lineno, file, line) > + else: > + import warnings > + s = warnings.formatwarning(message, category, filename, lineno, line) > + logger = getLogger("py.warnings") > + if not logger.handlers: > + logger.addHandler(NullHandler()) > + logger.warning("%s", s) I'd be careful here - this could deadlock if a thread spawned as a side effect of importing a module happens to trigger a warning. warnings is pulled into sys.modules as part of the interpreter startup - having a global "import warnings" shouldn't have any real effect on logging's import time. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

1 0

2.5.3 and 2.4.6 release schedule
by "Martin v. Löwis" 04 Dec '08

04 Dec '08

I would like to create 2.5.3 and 2.4.6 release candidates next week, December 12, and final releases on December 19. If there are any open issues that you think need to be considered, please create a bug in the bug tracker, mark it as release blocker, and label it with version 2.5.3 (or 2.4). Of course, a number of such issues are already in the tracker, some already being worked on. Remember: 2.5.3 will be the last bug fix release for Python 2.5; afterwards, only security patches will be accepted for the 2.5 branch. The 2.4 branch is already in that state (the 2.3 branch is not maintained anymore; 2.4 security patches will be produced until November 2009). Regards, Martin

1 0

Accessing source code in zipped packages
by Alexander Belopolsky 03 Dec '08

03 Dec '08

About a month ago, I submitted two patches that address Pdb and doctest inability to load source code from modules with custom loaders such as modules loaded from zip files: http://bugs.python.org/issue4201 http://bugs.python.org/issue4197 The patches are very simple, basically calls to linecache.getline() need to be provided with the module's dict to enable linecache to find the module's __loader__. Is there a chance that these patches could make it to 2.6.1?

3 2

Attribute error: providing type name
by Filip Gruszczyński 03 Dec '08

03 Dec '08

This is my first message in this list, therefore I would like to say hello to everyone. I also hope, that I am not breaking any rules or guidelines for sending proposals. I would like to ask, if it is possible to provide type name of the object invoking the exception, when Attribute error is catched. It is done for functions, like: AttributeError: 'function' object has no attribute 'getValue' but for some objects there is only: AttributeError: connectToBases This is cool, when you know exactly what type of object cast the exception. But if there might be many of them, you must do one of two things: add print statement just before the line with the exception and check the type or iterate over all classes that might appear them. Showing the class name would solve this problem and could save a lot of time. -- Filip Gruszczyński

9 18

Tomorrow's releases
by Barry Warsaw 02 Dec '08

02 Dec '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I believe we are on track for releasing Python 3.0 final and 2.6.1 tomorrow. There is just one release blocker for 3.0 left -- Guido needs to finish the What's New for 3.0. This is bug 2306. So that Martin can have something to work with when he wakes up tomorrow morning, I would like to tag and branch the tree some time today, Tuesday 02-Dec US/Eastern. Therefore I am freezing both the 2.6 and 3.0 trees, with special dispensation to Guido for the updated What's New. Ping me on irc @ freenode #python-dev if you have anything else to check in to either tree before then. As soon as I hear from Guido, or issue 2306 is closed, I'm branching 3.0 and tagging it for release. Great work everyone, we're almost there! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSTWbPnEjvBPtnXfVAQKtOgP9EZgGkE8/UY1IRn7j0l6vX6uqbPapg+9H MlBIZrA6mEbGiaDSvPRiwuo71jP5cg0u/xFRdDlGYl0GAzOEWvKCZVlVsndM4kbh 7UxHjHfkIOo4MUw4zz1NrJ4GRNgBQa52OOtiOKKkIhr/oMsg+GWv8Y9hRXYA9xue s8as2AQe2QU= =5j55 -----END PGP SIGNATURE-----

1 0

Move encoding_decl to the top of Grammar/Grammar?
by Thomas Lee 02 Dec '08

02 Dec '08

Hi all, Currently, Parser/parsetok.c has a dependency on graminit.h. This can cause headaches when rebuilding after adding new syntax to Grammar/Grammar because parsetok.c is part of pgen, which is responsible for *generating* graminit.h. This circular dependency can result in parsetok.c using a different value for encoding_decl to what is used in ast.c, which causes PyAST_FromNode to fall over at runtime. It effectively looks something like this: * Grammar/Grammar is modified * build begins -- pgen compiles, parsetok.c uses encoding_decl=X * graminit.h is rebuilt with encoding_decl=Y * ast.c is compiled using encoding_decl=Y * when python runs, parsetok() emits encoding_decl nodes that PyAST_FromNode can't recognize: SystemError: invalid node XXX for PyAST_FromNode A nice, easy short term solution that doesn't require unwinding this dependency would be to simply move encoding_decl to the top of Grammar/Grammar and add a big warning noting that it needs to come before everything else. This will help to ensure its value never changes when syntax is added/removed. I'm happy to provide a patch for this (including some additional dependency info for files dependent upon graminit.h and Python-ast.h), but was wondering if there were any opinions about how this should be resolved. Cheers, Tom

1 1

distutils doesn't use some compiler options when building
by Akira Kitada 01 Dec '08

01 Dec '08

Hi, I encountered a weird problem using distutils. Generally, distutils try to use the same compiler options used for building Python interpreter, but it looks like some of them are omitted sometimes. - CPPFLAGS are not retrieved from the config and only ones in env are used. - OPT is retrieved from the config, but it's only used when env has CFLAGS. See: http://svn.python.org/view/python/trunk/Lib/distutils/sysconfig.py """ def customize_compiler(compiler): ... if compiler.compiler_type == "unix": (cc, cxx, opt, cflags, ccshared, ldshared, so_ext) = \ get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'SO') if 'CC' in os.environ: cc = os.environ['CC'] if 'CXX' in os.environ: cxx = os.environ['CXX'] if 'LDSHARED' in os.environ: ldshared = os.environ['LDSHARED'] if 'CPP' in os.environ: cpp = os.environ['CPP'] else: cpp = cc + " -E" # not always if 'LDFLAGS' in os.environ: ldshared = ldshared + ' ' + os.environ['LDFLAGS'] if 'CFLAGS' in os.environ: cflags = opt + ' ' + os.environ['CFLAGS'] ldshared = ldshared + ' ' + os.environ['CFLAGS'] if 'CPPFLAGS' in os.environ: cpp = cpp + ' ' + os.environ['CPPFLAGS'] cflags = cflags + ' ' + os.environ['CPPFLAGS'] ldshared = ldshared + ' ' + os.environ['CPPFLAGS'] cc_cmd = cc + ' ' + cflags compiler.set_executables( preprocessor=cpp, compiler=cc_cmd, compiler_so=cc_cmd + ' ' + ccshared, compiler_cxx=cxx, linker_so=ldshared, linker_exe=cc) compiler.shared_lib_extension = so_ext """ Are these logics are intentional or just a bug? If this is intentional behavior, why is that being this way? Thanks,

10 31

format specification mini-language docs...
by Dino Viehland 01 Dec '08

01 Dec '08

Ok, now I'm implementing __format__ support for IronPython. The format spec mini-language docs say that a presentation type of None is the same as 'g' for floating point / decimal values. But these two formats seem to differ based upon how they handle whole numbers: >>> 2.0.__format__('') '2.0' >>> 2.0.__format__('g') '2' The docs also say that 'g' prints it as fixed point unless the number is too large. But the fixed point format differs from what 'f' would print. I guess it didn't say they'd both print it as fixed point w/ a precision of 6 or anything but it seems a little unclear. >>> 2.0.__format__('g') '2' >>> 2.0.__format__('f') '2.000000' Finally providing any sign character seems to cause +1.0#INF and friends to be returned instead of inf as is documented: >>> 10e667.__format__('+') '+1.0#INF' >>> 10e667.__format__('') 'inf' Are these just doc bugs? The inf issue is the only one that seems particularly weird to me.

2 4

Python under valgrind
by Amaury Forgeot d'Arc 01 Dec '08

01 Dec '08

Hrvoje Niksic wrote: > A friend pointed out that running python under valgrind (simply "valgrind > python") produces a lot of "invalid read" errors. Reading up on > Misc/README.valgrind only seems to describe why "uninitialized reads" should > occur, not invalid ones. For example: [...] > I suppose valgrind could be confused by PyFree's pool address validation > that intentionally reads the memory just before the allocated block, and > incorrectly attributes it to a previously allocated (and hence freed) block, > but I can't prove that. Has anyone investigated this kind of valgrind > report? Did you use the suppressions file as suggested in Misc/README.valgrind? --suppressions=Misc/valgrind-python.supp -- Amaury Forgeot d'Arc

4 3

Patch to speed up non-tracing case in PyEval_EvalFrameEx (2% on pybench)
by Jeffrey Yasskin 01 Dec '08

01 Dec '08

Tracing support shows up fairly heavily an a Python profile, even though it's nearly always turned off. The attached patch against the trunk speeds up PyBench by 2% for me. All tests pass. I have 2 questions: 1) Can other people corroborate this speedup on their machines? I'm running on a Macbook Pro (Intel Core2 processor, probably Merom) with a 32-bit build from Apple's gcc-4.0.1. (Apple's gcc consistently produces a faster python than gcc-4.3.) 2) Assuming this speeds things up for most people, should I check it in anywhere besides the trunk? I assume it's out for 3.0; is it in for 2.6.1 or 3.0.1? Pybench output: ------------------------------------------------------------------------------- PYBENCH 2.0 ------------------------------------------------------------------------------- * using CPython 2.7a0 (trunk:67458M, Nov 30 2008, 17:14:10) [GCC 4.0.1 (Apple Inc. build 5488)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.time ------------------------------------------------------------------------------- Benchmark: pybench.out ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-9.5.0-i386-32bit Processor: i386 Python: Implementation: CPython Executable: /Users/jyasskin/src/python/trunk-fast-tracing/build/python.exe Version: 2.7.0 Compiler: GCC 4.0.1 (Apple Inc. build 5488) Bits: 32bit Build: Nov 30 2008 17:14:10 (#trunk:67458M) Unicode: UCS2 ------------------------------------------------------------------------------- Comparing with: ../build_orig/pybench.out ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-9.5.0-i386-32bit Processor: i386 Python: Implementation: CPython Executable: /Users/jyasskin/src/python/trunk-fast-tracing/build_orig/python.exe Version: 2.7.0 Compiler: GCC 4.0.1 (Apple Inc. build 5488) Bits: 32bit Build: Nov 30 2008 13:51:09 (#trunk:67458) Unicode: UCS2 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 127ms 130ms -2.4% 129ms 132ms -2.1% BuiltinMethodLookup: 90ms 93ms -3.2% 91ms 94ms -3.1% CompareFloats: 88ms 91ms -3.3% 89ms 93ms -4.3% CompareFloatsIntegers: 97ms 99ms -2.1% 97ms 100ms -2.4% CompareIntegers: 79ms 82ms -4.2% 79ms 85ms -6.1% CompareInternedStrings: 90ms 92ms -2.4% 94ms 94ms -0.9% CompareLongs: 86ms 83ms +3.6% 87ms 84ms +3.5% CompareStrings: 80ms 82ms -3.1% 81ms 83ms -2.3% CompareUnicode: 103ms 105ms -2.3% 106ms 108ms -1.5% ComplexPythonFunctionCalls: 139ms 137ms +1.3% 140ms 139ms +0.1% ConcatStrings: 142ms 151ms -6.0% 156ms 154ms +1.1% ConcatUnicode: 87ms 92ms -5.4% 89ms 94ms -5.7% CreateInstances: 142ms 144ms -1.4% 144ms 145ms -1.1% CreateNewInstances: 107ms 109ms -2.3% 108ms 111ms -2.1% CreateStringsWithConcat: 114ms 137ms -17.1% 117ms 139ms -16.0% CreateUnicodeWithConcat: 92ms 101ms -9.2% 95ms 102ms -7.2% DictCreation: 77ms 81ms -4.4% 80ms 85ms -5.9% DictWithFloatKeys: 91ms 107ms -14.5% 93ms 109ms -14.6% DictWithIntegerKeys: 95ms 94ms +1.4% 108ms 96ms +12.3% DictWithStringKeys: 83ms 88ms -5.8% 84ms 88ms -4.7% ForLoops: 72ms 72ms -0.1% 79ms 74ms +5.8% IfThenElse: 83ms 80ms +3.9% 85ms 80ms +5.3% ListSlicing: 117ms 118ms -0.7% 118ms 121ms -1.8% NestedForLoops: 116ms 119ms -2.4% 121ms 121ms +0.0% NormalClassAttribute: 106ms 115ms -7.7% 108ms 117ms -7.7% NormalInstanceAttribute: 96ms 98ms -2.3% 97ms 100ms -3.1% PythonFunctionCalls: 92ms 95ms -3.7% 94ms 99ms -5.2% PythonMethodCalls: 147ms 147ms +0.1% 152ms 149ms +2.1% Recursion: 135ms 136ms -0.3% 140ms 144ms -2.9% SecondImport: 101ms 99ms +2.1% 103ms 101ms +2.2% SecondPackageImport: 107ms 103ms +3.5% 108ms 104ms +3.3% SecondSubmoduleImport: 134ms 134ms +0.3% 136ms 136ms -0.0% SimpleComplexArithmetic: 105ms 111ms -5.0% 110ms 112ms -1.4% SimpleDictManipulation: 95ms 106ms -10.6% 96ms 109ms -12.0% SimpleFloatArithmetic: 90ms 99ms -9.3% 93ms 102ms -8.2% SimpleIntFloatArithmetic: 78ms 76ms +2.3% 79ms 77ms +2.0% SimpleIntegerArithmetic: 78ms 77ms +1.8% 79ms 77ms +2.0% SimpleListManipulation: 80ms 78ms +2.4% 80ms 79ms +1.9% SimpleLongArithmetic: 110ms 113ms -2.0% 111ms 113ms -2.1% SmallLists: 128ms 117ms +9.5% 130ms 124ms +4.9% SmallTuples: 115ms 114ms +1.7% 117ms 114ms +2.2% SpecialClassAttribute: 101ms 112ms -10.3% 104ms 114ms -8.9% SpecialInstanceAttribute: 173ms 177ms -1.9% 176ms 179ms -1.6% StringMappings: 165ms 167ms -1.2% 168ms 169ms -0.5% StringPredicates: 126ms 134ms -5.7% 127ms 134ms -5.6% StringSlicing: 125ms 123ms +1.9% 131ms 130ms +0.7% TryExcept: 79ms 80ms -0.6% 80ms 80ms -0.8% TryFinally: 110ms 107ms +3.0% 111ms 112ms -1.1% TryRaiseExcept: 99ms 101ms -1.6% 100ms 102ms -1.7% TupleSlicing: 127ms 127ms +0.6% 137ms 137ms +0.0% UnicodeMappings: 144ms 144ms -0.3% 145ms 145ms -0.4% UnicodePredicates: 116ms 114ms +1.3% 117ms 115ms +1.1% UnicodeProperties: 106ms 102ms +3.6% 107ms 104ms +3.1% UnicodeSlicing: 95ms 111ms -14.0% 99ms 112ms -11.8% WithFinally: 157ms 152ms +3.3% 159ms 154ms +3.3% WithRaiseExcept: 123ms 125ms -1.1% 125ms 126ms -1.2% ------------------------------------------------------------------------------- Totals: 6043ms 6182ms -2.2% 6185ms 6301ms -1.9% (this=pybench.out, other=../build_orig/pybench.out) 2to3 times: Before: $ time ./python.exe ~/src/2to3/2to3 -f all ~/src/2to3/ >/dev/null real 0m56.685s user 0m55.620s sys 0m0.380s After: $ time ./python.exe ~/src/2to3/2to3 -f all ~/src/2to3/ >/dev/null real 0m55.067s user 0m53.843s sys 0m0.376s == 3% faster Gory details: The meat of the patch is: @@ -884,11 +891,12 @@ fast_next_opcode: f->f_lasti = INSTR_OFFSET(); /* line-by-line tracing support */ - if (tstate->c_tracefunc != NULL && !tstate->tracing) { + if (_Py_TracingPossible && + tstate->c_tracefunc != NULL && !tstate->tracing) { This converts the generated assembly (produced with `gcc -S -dA ...`, then manually annotated a bit) from: # basic block 17 # ../Python/ceval.c:885 LM541: movl 8(%ebp), %ecx LVL319: subl -316(%ebp), %edx movl %edx, 60(%ecx) # ../Python/ceval.c:889 LM542: # %esi = tstate movl -336(%ebp), %esi LVL320: # %eax = tstate->c_tracefunc movl 28(%esi), %eax LVL321: # if tstate->c_tracefunc == 0 testl %eax, %eax # goto past-if () je L567 # more if conditions here to: # basic block 17 # ../Python/ceval.c:889 LM542: movl 8(%ebp), %ecx LVL319: subl -316(%ebp), %edx movl %edx, 60(%ecx) # ../Python/ceval.c:893 LM543: # %eax = _Py_TracingPossible movl __Py_TracingPossible-"L00000000033$pb"(%ebx), %eax LVL320: # if _Py_TracingPossible != 0 testl %eax, %eax # goto rest-of-if (nearby) jne L2321 # opcode = NEXTOP(); continues here The branch should be predicted accurately either way, so there are 2 things that may be contributing to the performance change. First, adding the global caching variable halves the amount of memory that has to be read to check the prediction. The memory that is read is still read one instruction before it's used, but adding a local variable to read the memory earlier doesn't affect the performance. Without the global variable, the compiler puts the tracing code immediately after the if; with the global, it moves it away and puts the non-tracing code immediately after the first test in the if. This may affect branch prediction and may affect the icache. I tried using gcc's __builtin_expect() to ensure that the tracing code is always out-of-line. This moved it much farther away and cost about 1% in performance (i.e. 1% instead of 2% faster than "before"). I don't know why the __builtin_expect() version would be slower. If anyone feels inspired to test this out on another processor or compiler version, let me know how it goes. Jeffrey

2 2