Mailman 3 July 2010 - Python-Dev

How to spell PyInstance_NewRaw in py3k?
by Alexander Belopolsky July 15, 2010

July 15, 2010

Issue #5180 [1] presented an interesting challenge: how to unpickle instances of old-style classes when a pickle created with 2.x is loaded in 3.x python? The problem is that pickle protocol requires that unpickled instances be created without calling the __init__ method. This is necessary because pickle file may not contain information about how __init__ method should be invoked. Instead, implementations are required to bypass __init__ and populate instance's __dict__ directly using data found in the pickle. Pure python implementation uses the following trick that happens to work in 3.x: class Empty: pass pickled = Empty() pickled.__class__ = Pickled This of course, creates a new-style class in 3.x, but if 3.x version of Pickled behaves similarly to its 2.x predecessor, it should work. The cPickle implementation, on the other hand uses 2.x C API which is not available in 3.x. Namely, the PyInstance_NewRaw function. In order to fix the bug described in issue #5180, I had to emulate PyInstance_NewRaw using type->tp_alloc. I considered an rejected the idea to use tp_new instead. [2] Is this the right way to proceed? The patch is attached to the issue. [3] [1] http://bugs.python.org/issue5180 [2] http://bugs.python.org/issue5180#msg108846 [3] http://bugs.python.org/file17792/issue5180.diff

2 3

Re: [Python-Dev] notifications form Hg to python-checkins
by Nick Coghlan July 15, 2010

July 15, 2010

On Wed, Jul 14, 2010 at 5:56 AM, Éric Araujo <merwok(a)netwok.org> wrote: >> I was actually going to suggest something along those lines post >> transition, since Hg will have an email address for everyone - leave >> push notifications coming from the bot (so existing filters based on >> the bot address keep working), but include the push author in the >> reply-to field. > > IIRC, you can’t know who pushed without kludgy hackery. Also, the one to > push is not necessarily the author of the commit(s). If that was just a > braino and you meant setting the committer as reply-to, it seems a good > idea. No, I meant push. There's a separate discussion where it was pointed out that publishing each commit as a separate email makes python-checkins even chattier than it is already (this point came up after Tarek pushed a distutils2 changeset containing ~100 separate commits), so the suggestion was made to combine all commit notifications for a given push into a single email. However, I don't remember if that idea was suggested specifically to Dirkjan as a change to the notification email. If not, I'm suggesting it now :) Note that the current distutils2 emails to python-checkins already say "Tarek Ziade pushed..." and then include the committer details inside the email, and that's what we want - the developer pushing the change into the central repository is always going to be one of the core devs, but the original creator of the change set could have been anyone. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 6

str(memoryview('abc')) != 'abc' in Python 2.7
by M.-A. Lemburg July 15, 2010

July 15, 2010

Is this intended or should I open a bug report for it: >>> m = memoryview('abc') >>> m == 'abc' True >>> str(m) == 'abc' False >>> str(m) '<memory at 0x2b2bb6ee26d8>' I would have expected str(m) == 'abc'. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2010-07-19: EuroPython 2010, Birmingham, UK 3 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

4 4

Re: [Python-Dev] [Idle-dev] Removing IDLE from the standard library
by Bruce Sherwood July 15, 2010

July 15, 2010

Perhaps there are two separable issues. Many of us see it as extremely important that some IDLE be part of the standard Python distribution ("batteries included"), for the reasons that several people have given. However, there is merit to the suggestion to have an active separate development, with the understanding that periodically this separate development is a candidate for inclusion in the standard distribution, replacing whatever IDLE had been there. In the 2009 Google Summer of Code I was the mentor for a Brazilian student, Guilherme Polo, who completed and extended important improvements to IDLE made during the previous year by David Scherer. Given the somewhat official nature of this work, I assumed that these needed improvements would make it into the standard distribution, but as far as I know that hasn't happened. It would seem that if even this "sponsored" project didn't impact the standard Python distribution, something is broken in the procedures, and probably what is needed is, as Guido says, that someone be given the authority to get improvements to IDLE into the standard distribution. Making a significant change to the update procedures is clearly needed. Even if this needed change is made, there is also merit to Tai's suggestion of creating a separate project, to encourage developers like him to work together to improve IDLE, without having as a first priority to worry about getting it into the standard distribution, but with the clear understanding that this is the place to go for improvements to migrate into the standard distribution. Bruce Sherwood P.S. I'll add that IDLE has been EXTREMELY important for a large population of relatively casual users of Python, the thousands of science and engineering university students enrolled in the "Matter & Interactions" intro physics curriculum developed by Ruth Chabay and me ( matterandinteractions.org). A major feature of this curriculum is a serious introduction to computational modeling, in which students write short Python programs to model physical systems. Computational modeling is now central to all of science and engineering but alas has not made its way into undergraduate curricula in an institutionalized way. A big difficulty is that students come to college knowledgeable about all aspects of computers save one: programming. So the programming environment has to be exceptionally easy to learn and use. Python itself has the necessary properties, and Python+Visual (called VPython, vpython.org) lets the students focus on the physics while VPython generates real-time navigable 3D animations of the computational models, as a side effect of the computational code. IDLE has proved to be the right editing tool for this population, as essentially nothing needs to be learned, and there is near-instantaneous edit/run transitions which encourage rapid testing. In a physics course we have to focus on strict minimalism as far as the programming is concerned. We teach a bare minimum of needed concepts; for example, we introduce while loops but not for loops. We cannot afford to teach about a professional IDE; IDLE is fine and has worked well for us. (We're currently bundling with VPython the Scherer/Polo version of IDLE, which for reasons of clarity we're calling VIDLE.) A final personal note is that while I use Eclipse for working on the Visual module, which is written in C++, I find VIDLE a delightful environment for developing Python programs for physics education.

18 38

Unordered tuples/lists
by Gustavo Narea July 15, 2010

July 15, 2010

Hello, everybody. I've been searching for a data structure like a tuple/list *but* unordered -- like a set, but duplicated elements shouldn't be removed. I have not even found a recipe, so I'd like to write an implementation and contribute it to the "collections" module in the standard library. This is the situation I have: I have a data structure that represents an arithmetic/boolean operation. Operations can be commutative, which means that the order of their operands don't change the result of the operation. This is, the following operations are equivalent: operation(a, b, c) == operation(c, b, a) == operation(b, a, c) operation(a, b, a) == operation(a, a, b) == operation(b, a, a) operation(a, a) == operation(a, a) So, I need a type to store the arguments/operands so that if two of these collections have the same elements with the same multiplicity, they are equivalent, regardless of the order. A multiset is not exactly what I need: I still need to use the elements in the order they were given. For example, the logical conjuction (aka "and" operator) has a left and right operands; I need to evaluate the first/left one and, if it returned True, then call the second/right one. They must not be evaluated in a random order. To sum up, it would behave like a tuple or a list, except when it's compared with another object: They would be equivalent if they're both unordered tuples/lists, and have the same elements. There can be mutable and immutable editions (UnorderedList and UnorderedTuple, respectively). I will write a PEP to elaborate on this if you think it'd be nice to have. Or, should I have written the PEP first? Cheers, -- Gustavo Narea <xri://=Gustavo>. | Tech blog: =Gustavo/(+blog)/tech ~ About me: =Gustavo/about |

10 14

Issue 2986: difflib.SequenceMatcher is partly broken
by Terry Reedy July 15, 2010

July 15, 2010

[Also posted to http://bugs.python.org/issue2986 Developed with input from Eli Bendersky, who will write patchfile(s) for whichever change option is chosen.] Summary: difflib.SeqeunceMatcher was developed, documented, and originally operated as "a flexible class for comparing pairs of sequences of any [hashable] type". An "experimental" heuristic was added in 2.3a1 to speed up its application to sequences of code lines, which are selected from an unbounded set of possibilities. As explained below, this heuristic partly to completely disables SequenceMatcher for realistic-length sequences from a small finite alphabet. The regression is easy to fix. The docs were never changed to reflect the effect of the heuristic, but should be, with whatever additional change is made. In the commit message for revision 26661, which added the heuristic, Tim Peters wrote "While I like what I've seen of the effects so far, I still consider this experimental. Please give it a try!" Several people who have tried it discovered the problem with small alphabets and posted to the tracker. Issues #1528074, #1678339. #1678345, and #4622 are now-closed duplicates of #2986. The heuristic needs revision. Open questions (discussed after the examples): what exactly to do, which versions to do it too, and who will do it. --- Some minimal difference examples: from difflib import SequenceMatcher as SM # base example print(SM(None, 'x' + 'y'*199, 'y'*199).ratio()) # should be and is 0.9975 (rounded) # make 'y' junk print(SM(lambda c:c=='y', 'x' + 'y'*199, 'y'*199).ratio()) # should be and is 0.0 # Increment b by 1 char print(SM(None, 'x' + 'y'*199, 'y'*200).ratio()) # should be .995, but now is 0.0 because y is treated as junk # Reverse a and b, which increments b print(SM(None, 'y'*199, 'x' + 'y'*199).ratio()) # should be .9975, as before, but now is 0.0 because y is junked The reason for the bug is the heuristic: if the second sequence is at least 200 items long then any item occurring more than one percent of the time in the second sequence is treated as junk. This was aimed at recurring code lines like 'else:' and 'return', but can be fatal for small alphabets where common items are necessary content. A more realistic example than the above is comparing DNA gene sequences. Without the heuristic SequenceMatcher.get_opcodes() reports an appropriate sequence of matches and edits and .ratio works as documented and expected. For 1000/2000/6000 bases, the times on a old Athlon 2800 machine are <1/2/12 seconds. Since 6000 is longer than most genes, this is a realistic and practical use. With the heuristic, everything is junk and there is only one match, ''=='' augmented by the initial prefix of matching bases. This is followed by one edit: replace the rest of the first sequence with the rest of the second sequence. A much faster way to find the first mismatch would be i = 0 while first[i] == second[i]: i+=1 The match ratio, based on the initial matching prefix only, is spuriously low. --- Questions: 1: what change should be make. Proposed fix: Disentangle the heuristic from the calculation of the internal b2j dict that maps items to indexes in the second sequence b. Only apply the heuristic (or not) afterward. Version A: Modify the heuristic to only eliminate common items when there are more than, say, 100 items (when len(b2j)> 100 where b2j is first calculated without popularity deletions). The would leave DNA, protein, and printable ascii+[\n\r\t] sequences alone. On the other hand, realistic sequences of more than 200 code lines should have at least 100 different lines, and so the heuristic should continue to be applied when it (mostly?) 'should' be. This change leaves the API unchanged and does not require a user decision. Version B: add a parameter to .__init__ to make the heuristic optional. If the default were True ('use it'), then the code would run the same as now (even when bad). With the heuristic turned off, users would be able to get the .ratio they may expect and need. On the other hand, users would have to understand the heuristic to know when and when not to use it. Version C: A more radical alternative would be to make one or more of the tuning parameters user settable, with one setting turning it off. 2. What type of issue is this, and what version get changed. I see the proposal as partial reversion of a change that sometimes causes a regression, in order to fix the regression. Such would usually be called a bugfix. Other tracker reviewers claim this issue is a feature request, not a bugfix. Either way, 3.2 gets the fix. The practical issue is whether at least 2.7(.1) should get the fix, or whether the bug should forever continue in 2.x. 3. Who will make the change. Eli will write a patch and I will check it. However, Georg Brandel assigned the issue to Tim Peters, with a request for comment, but Tim never responded. Is there an active committer who will grab the issue and do a commit review when a patch is ready? -- Terry Jan Reedy

8 24

Removing IDLE from the standard library
by Tal Einat July 14, 2010

July 14, 2010

Hello, I would like to propose removing IDLE from the standard library. I have been using IDLE since 2002 and have been doing my best to help maintain and further develop IDLE since 2005. In recent years IDLE has received negligible interest and attention from the Python community. During this time IDLE has slowly gone downhill. The documentation and tutorials grow increasingly out of date. Cross-platform support has degraded with the increasing popularity of OSX and 64-bit platforms. Bugs take months, and sometimes more than a year, to be solved. Features that have since become common-place, such as having a non-intrusive search box instead of a dialog, are obviously and painfully lacking, making IDLE feel clumsy and out-dated. For these reasons, I think it would be fitting to remove IDLE from the standard library. IDLE is no longer recommended to beginners, IMO rightfully so, and this was the main reason for its inclusion in the standard library. Furthermore, if there is little or no interest in developing and maintaining IDLE, it should be removed to avoid having buggy and badly supported software in the standard library. - Tal Einat

33 79

Re: [Python-Dev] [Idle-dev] Removing IDLE from the standard library
by Bruce Sherwood July 13, 2010

July 13, 2010

On the notion that IDLE is fatally flawed and is driving away potential users of Python (to put the statements in their most extreme form): It seems that there are (at least) two very different communities people have in mind. I can appreciate that highly expert programmers may find IDLE insufficient for their needs, and it might even be the case that these people should be advised not to even try IDLE. But I want to stress as strongly as possible that for a very large community of nonexpert users of Python, IDLE has been extremely important and works very well for their purposes. Editing is pretty much like what they're already used to in word processors, which is certainly not the case with professional tools such as vim or Emacs. That said, yes, there are some significant deficiencies with the current IDLE (which is why there's a VIDLE, for instance). I'm very glad to hear from Martin that Guilherme is free to commit his important bug fixes and improvements. I'm afraid that neither Guilherme nor I understood the procedures and thought that we had to wait for others to act. Bruce Sherwood

15 40

How to block a module import
by Alexander Belopolsky July 13, 2010

July 13, 2010

I thought that in order to block a module from being imported, one would need to assign None to the corresponding entry in sys.modules. However, it looks like the code in test.support uses 0 instead of None: def _save_and_block_module(name, orig_modules): """Helper function to save and block a module in sys.modules Return value is True if the module was in sys.modules and False otherwise.""" saved = True try: orig_modules[name] = sys.modules[name] except KeyError: saved = False sys.modules[name] = 0 # <--- I would expect None here return saved In my experiments, 0 is not equivalent to None: >>> import sys, time >>> sys.modules['time'] = 0 >>> __import__('time') 0 >>> sys.modules['time'] = None >>> __import__('time') Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named time Am I missing something?

3 4

Threading bug review + commit privs
by Reid Kleckner July 13, 2010

July 13, 2010

Hey all, I'm porting some fixes for threading.py that we applied to unladen-swallow: http://bugs.python.org/issue6643 We ran into these bizarre race conditions involving fork + threads while running the test suite with a background JIT compilation thread. I really wish we weren't trying to support forking from a child thread, but it's already in the test suite. I've "solved" the problem by throwing away radioactive locks that may have been held across a fork.* If I could get a reviewer to look at this, I would be very grateful, since reviewing threading patches is somewhat tricky. =/ I'm also expecting to be doing more work merging unladen-swallow into the py3k-jit branch, so I was wondering if I could get commit privileges for that. Thanks, Reid * In general I wouldn't think this is safe, but we already do it for _active_limbo_lock in threading.py. One of the problems I've encountered is that on OS X, releasing locks held by other threads across a fork results in a crash. Furthermore, when locks are deallocated, the destructor does a non-blocking acquire and then release, which I would think would crash. However, we get lucky here, because any thread that holds a lock across a fork usually holds a reference to it on the stack. Therefore the lock is leaked and the destructor never run. Moral: fork + threads is *crazy*, avoid it if you can.

6 6