Mailman 3 February 2012 - Python-Dev

Re: [Python-Dev] [Python-checkins] cpython (3.2): remove unused import
by Brett Cannon 16 Feb '12

16 Feb '12

I'm going to assume pylint or pyflakes would throw too many warnings on the stdlib, but would it be worth someone's time to write a simple unused import checker to run over the stdlib on occasion? I bet even one that did nothing more than a regex search for matched import statements would be good enough. On Fri, Feb 3, 2012 at 19:09, benjamin.peterson <python-checkins(a)python.org>wrote: > http://hg.python.org/cpython/rev/9eb5fec8674b > changeset: 74749:9eb5fec8674b > branch: 3.2 > parent: 74746:5eb47e1732a0 > user: Benjamin Peterson <benjamin(a)python.org> > date: Fri Feb 03 19:07:30 2012 -0500 > summary: > remove unused import > > files: > Lib/threading.py | 1 - > 1 files changed, 0 insertions(+), 1 deletions(-) > > > diff --git a/Lib/threading.py b/Lib/threading.py > --- a/Lib/threading.py > +++ b/Lib/threading.py > @@ -5,7 +5,6 @@ > > from time import time as _time, sleep as _sleep > from traceback import format_exc as _format_exc > -from collections import deque > from _weakrefset import WeakSet > > # Note regarding PEP 8 compliant names > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > >

5 7

How to round timestamps and durations?
by Victor Stinner 15 Feb '12

15 Feb '12

Hi, My work on the PEP 410 tries to unify the code to manipulate timestamps. The problem is that I'm unable to decide how to round these numbers. Functions using a resolution of 1 second (e.g. time.mktime) expects rounding towards zero (ROUND_HALF_DOWN), as does int(float). Example: >>> time.mktime(time.localtime(-1.9)), time.mktime(time.localtime(1.9)) (-1.0, 1.0) datetime.datetime.fromtimestamp() rounds to nearest with ties going away from zero (ROUND_HALF_UP). Example: >>> datetime.datetime.fromtimestamp(-1.1e-6), datetime.datetime.fromtimestamp(1.1e-6) (datetime.datetime(1970, 1, 1, 0, 59, 59, 999999), datetime.datetime(1970, 1, 1, 1, 0, 0, 1)) >>> datetime.datetime.fromtimestamp(-1.9e-6), datetime.datetime.fromtimestamp(1.9e-6) (datetime.datetime(1970, 1, 1, 0, 59, 59, 999998), datetime.datetime(1970, 1, 1, 1, 0, 0, 2)) datetime.timedelta * float and datetime.timedelta / float rounds to nearest with ties going to nearest even integer (ROUND_HALF_EVEN), as does round(). Example: >>> [(datetime.timedelta(microseconds=x) / 2.0).microseconds for x in range(6)] [0, 0, 1, 2, 2, 2] Should I also support multiple rounding methods depending on the operation and of the Python function? Should we always use the same rounding method? Antoine pointed me that ROUND_HALF_UP can produce timestamps "in the future", which is especially visible when using a resolution of 1 second. I like this rounding method because it limits the loss of precision to an half unit: abs(rounded - timestamp) <= 0.5. But it can be "surprising". The rounding method should maybe be the same than int(float) (so ROUND_HALF_DOWN) to avoid surprising results for applications using int(time.time()) for example (I had such problem with rotated logs and test_logging). -- There is an issue on rounding timedelta: http://bugs.python.org/issue8860 Victor

3 2

PyPy 1.8 released
by Maciej Fijalkowski 14 Feb '12

14 Feb '12

============================ PyPy 1.8 - business as usual ============================ We're pleased to announce the 1.8 release of PyPy. As habitual this release brings a lot of bugfixes, together with performance and memory improvements over the 1.7 release. The main highlight of the release is the introduction of `list strategies`_ which makes homogenous lists more efficient both in terms of performance and memory. This release also upgrades us from Python 2.7.1 compatibility to 2.7.2. Otherwise it's "business as usual" in the sense that performance improved roughly 10% on average since the previous release. you can download the PyPy 1.8 release here: http://pypy.org/download.html .. _`list strategies`: http://morepypy.blogspot.com/2011/10/more-compact-lists-with-list-strategie… What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 1.8 and cpython 2.7.1`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work has been stalled, we would welcome a volunteer to handle that. .. _`pypy 1.8 and cpython 2.7.1`: http://speed.pypy.org Highlights ========== * List strategies. Now lists that contain only ints or only floats should be as efficient as storing them in a binary-packed array. It also improves the JIT performance in places that use such lists. There are also special strategies for unicode and string lists. * As usual, numerous performance improvements. There are many examples of python constructs that now should be faster; too many to list them. * Bugfixes and compatibility fixes with CPython. * Windows fixes. * NumPy effort progress; for the exact list of things that have been done, consult the `numpy status page`_. A tentative list of things that has been done: * multi dimensional arrays * various sizes of dtypes * a lot of ufuncs * a lot of other minor changes Right now the `numpy` module is available under both `numpy` and `numpypy` names. However, because it's incomplete, you have to `import numpypy` first before doing any imports from `numpy`. * New JIT hooks that allow you to hook into the JIT process from your python program. There is a `brief overview`_ of what they offer. * Standard library upgrade from 2.7.1 to 2.7.2. Ongoing work ============ As usual, there is quite a bit of ongoing work that either didn't make it to the release or is not ready yet. Highlights include: * Non-x86 backends for the JIT: ARMv7 (almost ready) and PPC64 (in progress) * Specialized type instances - allocate instances as efficient as C structs, including type specialization * More numpy work * Since the last release there was a significant breakthrough in PyPy's fundraising. We now have enough funds to work on first stages of `numpypy`_ and `py3k`_. We would like to thank again to everyone who donated. * It's also probably worth noting, we're considering donations for the Software Transactional Memory project. You can read more about `our plans`_ Cheers, The PyPy Team .. _`brief overview`: http://doc.pypy.org/en/latest/jit-hooks.html .. _`numpy status page`: http://buildbot.pypy.org/numpy-status/latest.html .. _`numpy status update blog report`: http://morepypy.blogspot.com/2012/01/numpypy-status-update.html .. _`numpypy`: http://pypy.org/numpydonate.html .. _`py3k`: http://pypy.org/py3donate.html .. _`our plans`: http://morepypy.blogspot.com/2012/01/transactional-memory-ii.html

3 2

Re: [Python-Dev] peps: Update with bugfix releases.
by Ned Deily 13 Feb '12

13 Feb '12

In article <E1Ru7G3-0007Mb-3R(a)dinsdale.python.org>, georg.brandl <python-checkins(a)python.org> wrote: > +Bugfix Releases > +=============== > + > +- 3.2.1: released July 10, 2011 > +- 3.2.2: released September 4, 2011 > + > +- 3.2.3: planned February 10-17, 2012 I would like to propose that we plan for 3.2.3 and 2.7.3 immediately after PyCon, so approximately March 17, if that works for all involved. My primary rationale is to allow time to address all of the OS X Xcode 4 issues for 10.6 and 10.7. They need to be fixed in 2.7.x, 3.2.x, and 3.3: right now it is not possible to build C extension modules in some sets of configurations. As I mentioned the other day, it is going to take a few more weeks to finish testing and generate all the fixes. -- Ned Deily, nad(a)acm.org

9 16

PEP 410 (Decimal timestamp): the implementation is ready for a review
by Victor Stinner 13 Feb '12

13 Feb '12

Hi, I finished the implementation of the PEP 410 ("Use decimal.Decimal type for timestamps"). The PEP: http://www.python.org/dev/peps/pep-0410/ The implementation: http://bugs.python.org/issue13882 Rietveld code review tool for this issue: http://bugs.python.org/review/13882/show The patch is huge because it changes many modules, but I plan to split the patch into small commits. I'm still waiting for Nick Coghlan and Guido van Rossum for their decision on the PEP. Victor

3 3

Adding a maximum element count to parse_qs?
by Antoine Pitrou 12 Feb '12

12 Feb '12

Hello, Given the randomization fix will ship disabled, I thought it would be nice to add a maximum element count argument to urlparse.parse_qs, with a default value of e.g. 1000 (including in bugfix releases). What do you think? Regards Antoine.

3 6

Fwd: maintenance of the ElementTree / cElementTree packages in the Python standard library
by Eli Bendersky 11 Feb '12

11 Feb '12

---------- Forwarded message ---------- From: Fredrik Lundh <fredrik(a)pythonware.com> Date: Fri, Feb 10, 2012 at 13:16 Subject: Re: maintenance of the ElementTree / cElementTree packages in the Python standard library To: Eli Bendersky <eliben(a)gmail.com> Hi Eli, thanks for reaching out. I'll get back to you with a more "formal" reply later, but yeah, that sounds like a plan -- I have very limited time for core Python work these days anyway (as you guys have probably noticed :-). But feel free to loop me in on suggested API changes going forward (and I'll dig up my notes on additions that would be nice to have if someone wants to work on that). </F> 2012/2/10 Eli Bendersky <eliben(a)gmail.com>: > Hello Fredrik, > > Recently a discussion came up on the python-dev mailing list regarding > continued maintenance of the ElementTree & cElementTree packages which > are part of the standard library, and which were originally > contributed by you. > > There currently exists an unclear situation with respect to the active > maintainer(s) of these packages. On one hand, PEP 360 states that the > packages are officially maintained in your repositories and all > problems should be assigned to you and fixed upstream. On the other > hand, it appears that there has already been made a considerable > amount of work (e.g. http://codereview.appspot.com/207048/show) solely > within the Python repositories, as well as the port to Python 3 which > now lives in all 3.x branches. In other words, de-facto the package > has been forked in the Python repository. Note that no changes (AFAIU) > have been made to the ElementTree *API*, only to the implementations > living in the stdlib. > > I'd like to understand your point of view on this topic. There are > currently 23 open issues on the package(s) in the Python tracker, and > some additional plans are being made (such as 'import ElementTree' > importing the C implementation by default, falling back on the Python > implementation if that's unavailable). Is that alright with you that > all such new fixes and developments are being made by Python code > developers in the Python repositories directly, without waiting for > your approval to submit them upstream? > > Thanks in advance, > Eli

3 2

Re: [Python-Dev] requirements for moving __import__ over to importlib?
by PJ Eby 11 Feb '12

11 Feb '12

On Feb 10, 2012 3:38 PM, "Brett Cannon" <brett(a)python.org> wrote: > On Fri, Feb 10, 2012 at 15:07, PJ Eby <pje(a)telecommunity.com> wrote: >> On Fri, Feb 10, 2012 at 1:05 PM, Brett Cannon <brett(a)python.org> wrote: >>> First is that if this were used on Windows or OS X (i.e. the OSs we support that typically have case-insensitive filesystems), then this approach would be a massive gain as we already call os.listdir() when PYTHONCASEOK isn't defined to check case-sensitivity; take your 5 stat calls and add in 5 listdir() calls and that's what you get on Windows and OS X right now. Linux doesn't have this check so you would still be potentially paying a penalty there. >> >> >> Wow. That means it'd always be a win for pre-stdlib sys.path entries, because any successful stdlib import equals a failed pre-stdlib lookup. (Of course, that's just saving some of the overhead that's been *added* by importlib, not a new gain, but still...) > > > How so? import.c does a listdir() as well (this is not special to importlib). IIRC, it does a FindFirstFile on Windows, which is not the same thing. That's one system call into a preallocated buffer, not a series of system calls and creation of Python string objects. > Don't care about automatic reloaders. I'm just asking about the case where the mtime granularity is coarse enough to allow for a directory change, an import to execute, and then another directory change to occur all within a single mtime increment. That would lead to the set cache to be out of date. Ah. Good point. Well, if there's any way to know what the mtime granularity is, we can avoid the race condition by never performing the listdir when the current clock time is too close to the stat(). In effect, we can bypass the optimization if the directory was just modified. Something like: mtime = stat(dir).st_mtime if abs(time.time()-mtime)>unsafe_window: old_mtime, files = cache.get(dir, (-1, ())) if mtime!=old_mtime: files = frozenset(listdir(dir)) cache[dir] = mtime, files # code to check for possibility of importing # and shortcut if found, or # exit with failure if no matching files # fallthrough to direct filesystem checking The "unsafe window" is presumably filesystem and platform dependent, but ISTR that even FAT filesystems have 2-second accuracy. The other catch is the relationship between st_mtime and time.time(); I assume they'd be the same in any sane system, but what if you're working across a network and there's clock skew? Ugh. Worst case example would be say, accessing a FAT device that's been shared over a Windows network from a machine whose clock is several hours off. So it always looks safe to read, even if it's just been changed. What's the downside in that case? You're trying to import something that just changed in the last fraction of a second... why? I mean, sure, the directory listing will be wrong, no question. But it only matters that it was wrong if you added, removed, or renamed importable files. Why are you trying to import one of them? Ah, here's a use case: you're starting up IDLE, and while it's loading, you save some .py files you plan to import later. Your editor saves them all at once, but IDLE does the listdir() midway through. You then do an import from the IDLE prompt, and it fails because the listdir() didn't catch everything. Okay, now I know how to fix this. The problem isn't that there's a race condition per se, the problem is that the race results in a broken cache later. After all, it could just as easily have been the case that the import failed due to timing. The problem is that all *future* imports would fail in this circumstance. So the fix is a time-to-live recheck: if TTL seconds have passed since the last use of the cached frozenset, reload it, and reset the TTL to infinity. In other words: mtime = stat(dir).st_mtime now - time.time() if abs(now-mtime)>unsafe_window: old_mtime, then, files = cache.get(dir, (-1, now, ())) if mtime!=old_mtime or then is not None and now-then>TTL: files = frozenset(listdir(dir)) cache[dir] = mtime, now if mtime!=old_mtime else None, files # code to check for possibility of importing # and shortcut if found, or # exit with failure if no matching files # fallthrough to direct filesystem checking What this does (or should do) is handle clock-skew race condition stale caches by reloading the listdir even if mtime hasn't changed, as soon as TTL seconds have passed since the last snapshot was taken. However, if the mtime stays the same, no subsequent listdirs will occur. As long as the TTL is set high enough that a full startup of Python can occur, but low enough that it resets by the time a human can notice something's wrong, it should be golden. ;-) The TTL approach could be used in place of the unsafe_window, actually; there's probably not much need for both. The pure unsafe_window approach has the advantage of elegance: it slows down only when you've just written to the directory, and only briefly. It doesn't load the directory twice, either. I suppose ideally, we'd set unsafe_window fairly low, and TTL fairly high, so that for command-line apps and such you'd be done your entire script (or at least all its importing) before reaching the TTL value. But interactive apps and servers wouldn't end up with a permanently stale cache just because something was changed during startup. Feh. Screw it, just use a fairly high TTL and forget trying to tune the unsafe_window, since if you're using a TTL you have to do the listdir() a second time if there are any imports later. It's also a single tunable parameter at that point. How high a TTL? It's got to be at least as high as the worst-case mtime granularity... which is how high? Yet, low enough that the human who goes, "huh, that import should've worked", checks the directory listing and tries it again will have it go through. Hopefully, the worst-case mtime granularity is shorter than that. ;-) >> Yep. I was actually thinking this could be backported to 2.x, even without importlib, as a module to be imported in sitecustomize or via a .pth file. All it needs is a path hook, after all, and a subclass of the pkgutil importer to test it. And if we can get some people with huge NFS libraries and/or zillions of .egg directories on sys.path to test it, we could find out whether it's a win, lose, or draw for those scenarios. > > > You can do that if you want, obviously I don't want to bother since it won't make it into Python 2.7. Of course. My thought wasn't to do this with a full version of importlib, just to make a proof-of-concept importer. All it really needs to do is skip over the normal import processing in the case where the cache says there's no way for the import to succeed; that's where all the speedup really comes from. On Feb 10, 2012 3:38 PM, "Brett Cannon" <brett(a)python.org> wrote:

2 1

Add a new "locale" codec?
by Victor Stinner 10 Feb '12

10 Feb '12

Hi, I added PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize() and PyUnicode_EncodeLocale() to Python 3.3 to fix bugs. I hesitate to expose this codec in Python: it can be useful is some cases, especially if you need to interact with C functions. The glib library has functions using the *current* locale encoding, g_locale_from_utf8() for example. Related issue with more information: http://bugs.python.org/issue13619 Victor

9 21

Re: [Python-Dev] requirements for moving __import__ over to importlib?
by PJ Eby 10 Feb '12

10 Feb '12

On Feb 9, 2012 9:58 AM, "Brett Cannon" <brett(a)python.org> wrote: > This actually depends on the type of ImportError. My current solution actually would trigger an ImportError at the import statement if no finder could locate the module. But if some ImportError was raised because of some other issue during load then that would come up at first use. That's not really a lazy import then, or at least not as lazy as what Mercurial or PEAK use for general lazy importing. If you have a lot of them, that module-finding time really adds up. Again, the goal is fast startup of command-line tools that only use a small subset of the overall framework; doing disk access for lazy imports goes against that goal.

8 15