Mailman 3 April 2012 - Python-Dev

The step command of pdb is broken
by Xavier de Gaye 05 May '12

05 May '12

Issue http://bugs.python.org/issue13183 raises the point that the step command of pdb is broken. This issue is 6 months old. A patch and test case have been proposed. The 'Lifecycle of a Patch' at http://docs.python.org/devguide/patch.html says <quote> If your patch has not received any notice from reviewers (i.e., no comment made) after a substantial amount of time then you may email python-dev(a)python.org asking for someone to take a look at your patch. </quote> I am the author of pyclewn, a Vim front end to pdb and gdb, and I would be grateful for any progress on this issue. The following pdb session shows the problem when running the three modules main.py, foo.py and bar.py. After the second step command, pdb does not stop (as it should) at lines foo.py:5 and foo.py:6, nor does it stop to print the return value of increment(). ================================================= main.py 1 import foo 2 3 result = foo.increment(100) 4 print('result', result) foo.py 1 import bar 2 3 def increment(arg): 4 v = bar.value() 5 result = arg + v 6 return result bar.py 1 def value(): 2 return 5 ================================================= $ python -m pdb main.py > /path_to/main.py(1)<module>() -> import foo (Pdb) import sys; sys.version '3.3.0a2+ (default:2c27093fd11f, Apr 30 2012, 10:51:35) \n[GCC 4.3.2]' (Pdb) break bar.py:2 Breakpoint 1 at /path_to/bar.py:2 (Pdb) continue > /path_to/bar.py(2)value() -> return 5 (Pdb) step --Return-- > /path_to/bar.py(2)value()->5 -> return 5 (Pdb) step > /path_to/main.py(4)<module>() -> print('result', result) (Pdb) ================================================= Xavier

5 8

Re: [Python-Dev] cpython: Issue #14428: Use the new time.perf_counter() and time.process_time() functions
by Georg Brandl 01 May '12

01 May '12

On 29.04.2012 03:04, victor.stinner wrote: > http://hg.python.org/cpython/rev/bd195749c0a2 > changeset: 76599:bd195749c0a2 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Sun Apr 29 03:01:20 2012 +0200 > summary: > Issue #14428: Use the new time.perf_counter() and time.process_time() functions [...] > diff --git a/Lib/timeit.py b/Lib/timeit.py > --- a/Lib/timeit.py > +++ b/Lib/timeit.py > @@ -15,8 +15,8 @@ > -n/--number N: how many times to execute 'statement' (default: see below) > -r/--repeat N: how many times to repeat the timer (default 3) > -s/--setup S: statement to be executed once initially (default 'pass') > - -t/--time: use time.time() (default on Unix) > - -c/--clock: use time.clock() (default on Windows) > + -t/--time: use time.time() > + -c/--clock: use time.clock() Does it make sense to keep the options this way? IMO the distinction should be to use either perf_counter() or process_time(), and the options could implement this (-t -> perf_counter, -c -> process_time). Georg

2 2

Re: [Python-Dev] [Python-checkins] cpython (3.2): #14236: fix docs for \S.
by Senthil Kumaran 30 Apr '12

30 Apr '12

On Sun, Apr 29, 2012 at 12:37:25PM +0200, ezio.melotti wrote: > range of Unicode whitespace characters. > - \S Matches any non-whitespace character; equiv. to [^ \t\n\r\f\v]. > + \S Matches any non-whitespace character; equivalent to [^\s]. Is this correct? While I understand what meant (or implied) \s is not a valid ascii character in the documentation we denoted the sets using ascii characters only. -- Senthil

1 0

suggestion regarding the contributor agreement form
by Tshepang Lekhonkhobe 30 Apr '12

30 Apr '12

Hi, It's not very obvious that printing this page http://www.python.org/psf/contrib/contrib-form/ actually prints only the form. Can you rather offer a downloadable image/pdf. As an aside, on Chromium, it appears on 2 separate pages, when there's enough space on the first.

1 0

Re: [Python-Dev] [Python-checkins] devguide: Record Richard Oudkerk.
by Eric V. Smith 30 Apr '12

30 Apr '12

> +- Richard Oudkerk was given push privileges on Apr 29 2012 by Antoine Pitrou > + on recommendation by Charles-François Natali and Jesse Noller, for various > + contributions to multiprocessing (and original authorship of > + multiprocessing's predecessor, the processing package). Could one of you (Antoine, Charles-Francois, or Jesse) ask Richard to subscribe to python-committers? Or if you're reading this, Richard, could you subscribe? It's at http://mail.python.org/mailman/listinfo/python-committers I think there may have been some other recent committers for whom I didn't see subscribe requests, but I don't track it all that closely. Eric.

1 0

[RFC] PEP 418: Add monotonic time, performance counter and process time functions
by Victor Stinner 29 Apr '12

29 Apr '12

Hi, Here is a simplified version of the first draft of the PEP 418. The full version can be read online. http://www.python.org/dev/peps/pep-0418/ The implementation of the PEP can be found in this issue: http://bugs.python.org/issue14428 I post a simplified version for readability and to focus on changes introduced by the PEP. Removed sections: Existing Functions, Deprecated Function, Glossary, Hardware clocks, Operating system time functions, System Standby, Links. --- PEP: 418 Title: Add monotonic time, performance counter and process time functions Version: f2bb3f74298a Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012) Author: Cameron Simpson <cs(a)zip.com.au>, Jim Jewett <jimjjewett(a)gmail.com>, Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-March-2012 Python-Version: 3.3 Abstract ======== This PEP proposes to add ``time.get_clock_info(name)``, ``time.monotonic()``, ``time.perf_counter()`` and ``time.process_time()`` functions to Python 3.3. Rationale ========= If a program uses the system time to schedule events or to implement a timeout, it will not run events at the right moment or stop the timeout too early or too late when the system time is set manually or adjusted automatically by NTP. A monotonic clock should be used instead to not be affected by system time updates: ``time.monotonic()``. To measure the performance of a function, ``time.clock()`` can be used but it is very different on Windows and on Unix. On Windows, ``time.clock()`` includes time elapsed during sleep, whereas it does not on Unix. ``time.clock()`` precision is very good on Windows, but very bad on Unix. The new ``time.perf_counter()`` function should be used instead to always get the most precise performance counter with a portable behaviour (ex: include time spend during sleep). To measure CPU time, Python does not provide directly a portable function. ``time.clock()`` can be used on Unix, but it has a bad precision. ``resource.getrusage()`` can also be used on Unix, but it requires to get fields of a structure and compute the sum of time spent in kernel space and user space. The new ``time.process_time()`` function acts as a portable counter that always measures CPU time (doesn't include time elapsed during sleep) and has the best available precision. Each operating system implements clocks and performance counters differently, and it is useful to know exactly which function is used and some properties of the clock like its resolution and its precision. The new ``time.get_clock_info()`` function gives access to all available information of each Python time function. New functions: * ``time.monotonic()``: timeout and scheduling, not affected by system clock updates * ``time.perf_counter()``: benchmarking, most precise clock for short period * ``time.process_time()``: profiling, CPU time of the process Users of new functions: * time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess, telnet and threading modules to implement timeout * time.perf_counter(): trace and timeit modules, pybench program * time.process_time(): profile module * time.get_clock_info(): pybench program to display information about the timer like the precision or the resolution The ``time.clock()`` function is deprecated because it is not portable: it behaves differently depending on the operating system. ``time.perf_counter()`` or ``time.process_time()`` should be used instead, depending on your requirements. ``time.clock()`` is marked as deprecated but is not planned for removal. Python functions ================ New Functions ------------- time.get_clock_info(name) ^^^^^^^^^^^^^^^^^^^^^^^^^ Get information on the specified clock. Supported clock names: * ``"clock"``: ``time.clock()`` * ``"monotonic"``: ``time.monotonic()`` * ``"perf_counter"``: ``time.perf_counter()`` * ``"process_time"``: ``time.process_time()`` * ``"time"``: ``time.time()`` Return a dictionary with the following keys: * Mandatory keys: * ``"implementation"`` (str): name of the underlying operating system function. Examples: ``"QueryPerformanceCounter()"``, ``"clock_gettime(CLOCK_REALTIME)"``. * ``"resolution"`` (float): resolution in seconds of the clock. * ``"is_monotonic"`` (bool): True if the clock cannot go backward. * Optional keys: * ``"precision"`` (float): precision in seconds of the clock reported by the operating system. * ``"is_adjusted"`` (bool): True if the clock is adjusted (e.g. by a NTP daemon). time.monotonic() ^^^^^^^^^^^^^^^^ Monotonic clock, i.e. cannot go backward. It is not affected by system clock updates. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid and is a number of seconds. On Windows versions older than Vista, ``time.monotonic()`` detects ``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7 days): it increases a delta by 2\ :sup:`32` each time than an overflow is detected. The delta is stored in the process-local state and so the value of ``time.monotonic()`` may be different in two Python processes running for more than 49 days. On more recent versions of Windows and on other operating systems, ``time.monotonic()`` is system-wide. Availability: Windows, Mac OS X, Unix, Solaris. Not available on GNU/Hurd. Pseudo-code [#pseudo]_:: if os.name == 'nt': # GetTickCount64() requires Windows Vista, Server 2008 or later if hasattr(time, '_GetTickCount64'): def monotonic(): return _time.GetTickCount64() * 1e-3 else: def monotonic(): ticks = _time.GetTickCount() if ticks < monotonic.last: # Integer overflow detected monotonic.delta += 2**32 monotonic.last = ticks return (ticks + monotonic.delta) * 1e-3 monotonic.last = 0 monotonic.delta = 0 elif os.name == 'mac': def monotonic(): if monotonic.factor is None: factor = _time.mach_timebase_info() monotonic.factor = timebase[0] / timebase[1] return _time.mach_absolute_time() * monotonic.factor monotonic.factor = None elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"): def monotonic(): return time.clock_gettime(time.CLOCK_HIGHRES) elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"): def monotonic(): return time.clock_gettime(time.CLOCK_MONOTONIC) On Windows, ``QueryPerformanceCounter()`` is not used even though it has a better precision than ``GetTickCount()``. It is not reliable and has too many issues. time.perf_counter() ^^^^^^^^^^^^^^^^^^^ Performance counter with the highest available precision to measure a duration. It does include time elapsed during sleep and is system-wide. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid and is a number of seconds. Pseudo-code:: def perf_counter(): if perf_counter.use_performance_counter: if perf_counter.performance_frequency is None: try: perf_counter.performance_frequency = _time.QueryPerformanceFrequency() except OSError: # QueryPerformanceFrequency() fails if the installed # hardware does not support a high-resolution performance # counter perf_counter.use_performance_counter = False else: return _time.QueryPerformanceCounter() / perf_counter.performance_frequency else: return _time.QueryPerformanceCounter() / perf_counter.performance_frequency if perf_counter.use_monotonic: # The monotonic clock is preferred over the system time try: return time.monotonic() except OSError: perf_counter.use_monotonic = False return time.time() perf_counter.use_performance_counter = (os.name == 'nt') if perf_counter.use_performance_counter: perf_counter.performance_frequency = None perf_counter.use_monotonic = hasattr(time, 'monotonic') time.process_time() ^^^^^^^^^^^^^^^^^^^ Sum of the system and user CPU time of the current process. It does not include time elapsed during sleep. It is process-wide by definition. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid. It is available on all platforms. Pseudo-code [#pseudo]_:: if os.name == 'nt': def process_time(): handle = win32process.GetCurrentProcess() process_times = win32process.GetProcessTimes(handle) return (process_times['UserTime'] + process_times['KernelTime']) * 1e-7 else: import os try: import resource except ImportError: has_resource = False else: has_resource = True def process_time(): if process_time.use_process_cputime: try: return time.clock_gettime(time.CLOCK_PROCESS_CPUTIME_ID) except OSError: process_time.use_process_cputime = False if process_time.use_getrusage: try: usage = resource.getrusage(resource.RUSAGE_SELF) return usage[0] + usage[1] except OSError: process_time.use_getrusage = False if process_time.use_times: try: times = os.times() return times[0] + times[1] except OSError: process_time.use_getrusage = False return _time.clock() process_time.use_process_cputime = ( hasattr(time, 'clock_gettime') and hasattr(time, 'CLOCK_PROCESS_CPUTIME_ID')) process_time.use_getrusage = has_resource # On OS/2, only the 5th field of os.times() is set, others are zeros process_time.use_times = (hasattr(os, 'times') and os.name != 'os2') Alternatives: API design ======================== Other names for time.monotonic() -------------------------------- * time.counter() * time.metronomic() * time.seconds() * time.steady(): "steady" is ambiguous: it means different things to different people. For example, on Linux, CLOCK_MONOTONIC is adjusted. If we uses the real time as the reference clock, we may say that CLOCK_MONOTONIC is steady. But CLOCK_MONOTONIC gets suspended on system suspend, whereas real time includes any time spent in suspend. * time.timeout_clock() * time.wallclock(): time.monotonic() is not the system time aka the "wall clock", but a monotonic clock with an unspecified starting point. The name "time.try_monotonic()" was also proposed for an older proposition of time.monotonic() which was falling back to the system time when no monotonic clock was available. Other names for time.perf_counter() ----------------------------------- * time.hires() * time.highres() * time.timer() Only expose operating system clocks ----------------------------------- To not have to define high-level clocks, which is a difficult task, a simpler approach is to only expose operating system clocks. time.clock_gettime() and related clock identifiers were already added to Python 3.3 for example. time.monotonic(): Fallback to system time ----------------------------------------- If no monotonic clock is available, time.monotonic() falls back to the system time. Issues: * It is hard to define correctly such function in the documentation: is it monotonic? Is it steady? Is it adjusted? * Some user want to decide what to do when no monotonic clock is available: use another clock, display an error, or do something else? Different APIs were proposed to define such function. One function with a flag: time.monotonic(fallback=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * time.monotonic(fallback=True) falls back to the system time if no monotonic clock is available or if the monotonic clock failed. * time.monotonic(fallback=False) raises OSError if monotonic clock fails and NotImplementedError if the system does not provide a monotonic clock A keyword argument that gets passed as a constant in the caller is usually poor API. Raising NotImplementedError for a function is something uncommon in Python and should be avoided. One time.monotonic() function, no flag ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ time.monotonic() returns (time: float, is_monotonic: bool). An alternative is to use a function attribute: time.monotonic.is_monotonic. The attribute value would be None before the first call to time.monotonic(). Choosing the clock from a list of constraints --------------------------------------------- The PEP as proposed offers a few new clocks, but their guarentees are deliberately loose in order to offer useful clocks on different platforms. This inherently embeds policy in the calls, and the caller must thus choose a policy. The "choose a clock" approach suggests an additional API to let callers implement their own policy if necessary by making most platform clocks available and letting the caller pick amongst them. The PEP's suggested clocks are still expected to be available for the common simple use cases. To do this two facilities are needed: an enumeration of clocks, and metadata on the clocks to enable the user to evaluate their suitability. The primary interface is a function make simple choices easy: the caller can use ``time.get_clock(*flags)`` with some combination of flags. This include at least: * time.MONOTONIC: clock cannot go backward * time.STEADY: clock rate is steady * time.ADJUSTED: clock may be adjusted, for example by NTP * time.HIGHRES: clock with the highest precision It returns a clock object with a .now() method returning the current time. The clock object is annotated with metadata describing the clock feature set; its .flags field will contain at least all the requested flags. time.get_clock() returns None if no matching clock is found and so calls can be chained using the or operator. Example of a simple policy decision:: T = get_clock(MONOTONIC) or get_clock(STEADY) or get_clock() t = T.now() The available clocks always at least include a wrapper for ``time.time()``, so a final call with no flags can always be used to obtain a working clock. Example of flags of system clocks: * QueryPerformanceCounter: MONOTONIC | HIGHRES * GetTickCount: MONOTONIC | STEADY * CLOCK_MONOTONIC: MONOTONIC | STEADY (or only MONOTONIC on Linux) * CLOCK_MONOTONIC_RAW: MONOTONIC | STEADY * gettimeofday(): (no flag) The clock objects contain other metadata including the clock flags with additional feature flags above those listed above, the name of the underlying OS facility, and clock precisions. ``time.get_clock()`` still chooses a single clock; an enumeration facility is also required. The most obvious method is to offer ``time.get_clocks()`` with the same signature as ``time.get_clock()``, but returning a sequence of all clocks matching the requested flags. Requesting no flags would thus enumerate all available clocks, allowing the caller to make an arbitrary choice amongst them based on their metadata. Example partial implementation: `clockutils.py <http://hg.python.org/peps/file/tip/pep-0418/clockutils.py>`_. Working around operating system bugs? ------------------------------------- Should Python ensure manually that a monotonic clock is truly monotonic by computing the maximum with the clock value and the previous value? Since it's relatively straightforward to cache the last value returned using a static variable, it might be interesting to use this to make sure that the values returned are indeed monotonic. * Virtual machines provide less reliable clocks. * QueryPerformanceCounter() has known bugs (only one is not fixed yet) Python may only work around a specific known operating system bug: `KB274323`_ contains a code example to workaround the bug (use GetTickCount() to detect QueryPerformanceCounter() leap). Issues of a hacked monotonic function: * if the clock is accidentally set forward by an hour and then back again, you wouldn't have a useful clock for an hour * the cache is not shared between processes so different processes wouldn't see the same clock value

15 52

package imports, sys.path and os.chdir()
by Christian Tismer 29 Apr '12

29 Apr '12

Howdy, I have a small problem/observation with imports. I have several packages to import, which works all fine, as long as the packages are imported from directories found on the installed site-packages, via .pth etc. The only problem is the automatically prepended empty string in sys.path. Depending from where I start my application, the values stored in package.__file__ and package.__path__ are absolute or relative paths. So, if my pwd is the directory that contains my top-level modules, even though sys.path contains correct absolute entries for that, in this case the '' entry wins. Assume this: <- cwd is here moda modb >>> import moda Some code happens to chdir away, and later some code does >>> from moda import modb Since the __path__ entry is now a relative path, this second import fails. Although it is no recommended practice to leave a changed chdir(), I don't see why this is so. When a module is imported, would it not be better to always make __file__ and __path__ absolute? I see the module path, hidden by the '' entry not as a feature but an undesired side-effect. No big deal and easy to work around, I just would like to understand why. cheers -- chris -- Christian Tismer :^)<mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a. PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

9 17

questions about memory management
by Julia Lawall 29 Apr '12

29 Apr '12

In Python-3.2.3/Python/import.c, in the function _PyImport_FixupExtensionUnicode, is any call to PyDict_DelItemString needed before the final failure returns? modules = PyImport_GetModuleDict(); if (PyDict_SetItemString(modules, name, mod) < 0) return -1; if (_PyState_AddModule(mod, def) < 0) { PyDict_DelItemString(modules, name); return -1; } if (def->m_size == -1) { if (def->m_base.m_copy) { /* Somebody already imported the module, likely under a different name. XXX this should really not happen. */ Py_DECREF(def->m_base.m_copy); def->m_base.m_copy = NULL; } dict = PyModule_GetDict(mod); if (dict == NULL) return -1; def->m_base.m_copy = PyDict_Copy(dict); if (def->m_base.m_copy == NULL) return -1; } In Python-3.2.3/Modules/ossaudiodev.c, in the function build_namelists, is it intentional that labels is not freed in the last failure case: if (PyModule_AddObject(module, "control_labels", labels) == -1) goto error2; if (PyModule_AddObject(module, "control_names", names) == -1) goto error1; return 0; error2: Py_XDECREF(labels); error1: Py_XDECREF(names); return -1; thanks, julia

2 1

Re: [Python-Dev] cpython (2.7): Issue #14448: mention pytz; patch by Andrew Svetlov
by Georg Brandl 28 Apr '12

28 Apr '12

On 24.04.2012 19:48, sandro.tosi wrote: > http://hg.python.org/cpython/rev/e0e421133d0f > changeset: 76532:e0e421133d0f > branch: 2.7 > parent: 76527:22767284de99 > user: Sandro Tosi <sandro.tosi(a)gmail.com> > date: Tue Apr 24 19:43:33 2012 +0200 > summary: > Issue #14448: mention pytz; patch by Andrew Svetlov > > files: > Doc/library/datetime.rst | 8 ++++++++ > 1 files changed, 8 insertions(+), 0 deletions(-) > > > diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst > --- a/Doc/library/datetime.rst > +++ b/Doc/library/datetime.rst > @@ -1521,6 +1521,14 @@ > other fixed-offset :class:`tzinfo` subclass (such as a class representing only > EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). > > +.. seealso:: > + > + `pytz <http://pypi.python.org/pypi/pytz/>`_ > + The Standard Library has no :class:`tzinfo` instances except for UTC, but ^^^^^^^^^^^^^^^^ we don't capitalize "standard library" > + it exists a third-party library which brings Olson timezone database to ^^ there ^ the Also, I'm not sure everybody knows what the "Olson database" is, so maybe that should be explained too. cheers, Georg

3 6

Changes in html.parser may cause breakage in client code
by Vinay Sajip 28 Apr '12

28 Apr '12

Following recent changes in html.parser, the Python 3 port of Django I'm working on has started failing while parsing HTML. The reason appears to be that Django uses some module-level data in html.parser, for example tagfind, which is a regular expression pattern. This has changed recently (Ezio changed it in ba4baaddac8d). Now tagfind (and other such patterns) are not marked as private (though not documented), but should they be? The following script (tagfind.py): import html.parser as Parser data = '<select name="stuff">' m = Parser.tagfind.match(data, 1) print('%r -> %r' % (Parser.tagfind.pattern, data[1:m.end()])) gives different results on 3.2 and 3.3: $ python3.2 tagfind.py '[a-zA-Z][-.a-zA-Z0-9:_]*' -> 'select' $ python3.3 tagfind.py '([a-zA-Z][-.a-zA-Z0-9:_]*)(?:\\s|/(?!>))*' -> 'select ' The trailing space later causes a mismatch with the end tag, and leads to the errors. Django's use of the tagfind pattern is in a subclass of HTMLParser, in an overridden parse_startag method. Do we need to indicate more strongly that data like tagfind are private? Or has the change introduced inadvertent breakage, requiring a fix in Python? Regards, Vinay Sajip

7 8