Mailman 3 June 2008 - Python-Dev

Progress on switching Windows build to Berkeley DB 4.7.25...
by Trent Nelson 19 Jul '08

19 Jul '08

Hi all, Jesus, apologies that this has taken so long for me to get back to, I've been completely and utterly swamped with client work the past few weeks. However, thanks to a couple of hours spare at Detroit airport yesterday, I was finally able to make some progress on updating the Windows Berkeley DB build to 4.7.25. I've checked in the work I've done so far to branches/tnelson-trunk-bsddb-47-upgrade. One thing I wanted to double check with you is the following change: Modified: python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py ============================================================================== --- python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py (original) +++ python/branches/tnelson-trunk-bsddb-47-upgrade/Lib/bsddb/test/test_replication.py Wed Jun 18 06:13:44 2008 @@ -94,7 +94,7 @@ # The timeout is necessary in BDB 4.5, since DB_EVENT_REP_STARTUPDONE # is not generated if the master has no new transactions. # This is solved in BDB 4.6 (#15542). - timeout = time.time()+2 + timeout = time.time()+10 while (time.time()<timeout) and not (self.confirmed_master and self.client_startupdone) : time.sleep(0.02) if db.version() >= (4,6) : Basically, when using +2, the test failed every so often when running the entire test_bsddb3 suite. I picked 10 arbitrarily; it improves things, but it's still not 100%, I still encounter the following failure every so often: ====================================================================== ERROR: test01_basic_replication (bsddb.test.test_replication.DBReplicationManager) ---------------------------------------------------------------------- Traceback (most recent call last): File "s:\src\svn+ssh\pythondev@svn.python.org\python\branches\tnelson-trunk-bsddb-47-upgrade\lib\bsddb\test\test_replication.py", line 101, in setUp self.assertTrue(time.time()<timeout) AssertionError Can you comment on this? Apart from this small issue, the other 311 tests pass on x86 and x64 with flying colours, so nice work, whatever you've been doing ;-) Regards, Trent.

2 1

unittest's redundant assertions: asserts vs. failIf/Unlesses
by Gabriel Grant 16 Jul '08

16 Jul '08

Hi all, This gem from unittest.py is pretty much the opposite of "one obvious way": # Synonyms for assertion methods assertEqual = assertEquals = failUnlessEqual assertNotEqual = assertNotEquals = failIfEqual assertAlmostEqual = assertAlmostEquals = failUnlessAlmostEqual assertNotAlmostEqual = assertNotAlmostEquals = failIfAlmostEqual assertRaises = failUnlessRaises assert_ = assertTrue = failUnless assertFalse = failIf Could these be removed for 3k? There was a short discussion about this among some of those those present in the Python Core sprint room at PyCon today and most preferred the "assertEqual" form for [Not][Almost]Equal and Raises. With assertFalse vs. failIf (and assertTrue vs. failUnless) there was far less agreement. JUnit uses assertTrue exclusively, and most people said they feel that using assertTrue would be more consistent, but many (myself included) still think failUnless and failIf are much more natural. Another issue with assertTrue is that it doesn't actually test for 'True', strictly speaking, since it is based on equality, not identity. Its also interesting to note the original commit message: > r34209 | purcell | 2003-09-22 06:08:12 -0500 (Mon, 22 Sep 2003) > [...] > - New assertTrue and assertFalse aliases for comfort of JUnit users > [...] assertEqual (and its cousins) were already present at that point. In any case, if the decision is made to not use failUnless, something still needs to be done with assert_ vs. assertTrue. assert_ seems somewhat better to me, in that it has fewer characters, but I think that a case could certainly be made to keep both of these. I certainly don't have the authority to make a call on any of this, but if someone else decides what colour to paint this bike shed, I can try to get it done (hopefully with 2.6 warnings) tomorrow. Cheers, -Gabriel P.S. If you were in the sprint room and feel terribly misrepresented, please feel free to give me a swift kick both on-list and in person tomorrow morning.

25 52

Assignment to None
by Curt Hagenlocher 14 Jul '08

14 Jul '08

My apologies if this is one of those "dead horse" issues. The following seems a little inconsistent to me: >>> c = C() >>> c.None Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: C instance has no attribute 'None' >>> c.None = 'foo' File "<stdin>", line 1 SyntaxError: assignment to None >>> setattr(c, 'None', 'foo') >>> c.None 'foo' >>> So, it's okay to setattr the attribute name "None" but not okay to set it directly? Is this deliberate or is it an unintentional side effect of parser changes to prevent assignment to None? -- Curt Hagenlocher curt(a)hagenlocher.org

14 27

Community buildbots and Python release quality metrics
by glyph＠divmod.com 09 Jul '08

09 Jul '08

Today on planetpython.org, Doug Hellman announced the June issue of Python magazine. The cover story this month is about Pybots, "the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible". Last week, there was a "beta" release of Python which, according to the community buildbots, cannot run any existing python software. Normally I'd be complaining here about Twisted, but in fact Twisted is doing relatively well right now; only 80 failing tests. Django apparently cannot even be imported. The community buildbots have been in a broken state for months now[1]. I've been restraining myself from whinging about this, but now that it's getting close to release, it's time to get these into shape, or it's time to get rid of them. There are a few separate issues here. One is, to what extent should Python actually be compatible between releases? There's not much point in saying "this release is compatible with python 2.5" if all python 2.5 programs need to be modified in order to run on python 2.6. Of course, it's quite likely that there are some 3rd-party Python programs that continue to work, but python.org provides links to 2 projects that don't and none that do. Another way to phrase this question is, "whose responsibility is it to make Python 2.5 programs run on Python 2.6"? Or, "what happens when the core team finds out that a change they have made has broken some python software 'in the wild'"? Here are a couple of ways to answer these questions: 1) It's the python core team's responsibility. There should be Python buildbots for lots of different projects and they should all be green before the code can be considered [one of: allowed in trunk, alpha, beta, RC]. 2) It's the python core team's responsibility as long as the major projects are using public APIs. The code should not be considered [alpha, beta, RC] if there are any known breakages in public APIs. 3) It's only the python core team's responsibility to notify projects of incompatible changes of which they are aware, which may occur in public or private APIs. Major projects with breakages will be given a grace period before a [beta, RC, final] to release a forward-compatible version. 4) The status quo: every new Python release can (and will, at least speaking in terms of 2.6) break lots of popular software, with no warning aside from the beta period. There are no guarantees. For obvious reasons, I'd prefer #1, but any of the above is better than #4. I don't think #4 is an intentional choice, either; I think it's the result of there being no clear consequence for community buildbots failing. Or, for that matter, of core buildbots failing. Investigating the history is tricky (I see a bunch of emails from Barry here, but I'm not sure which is the final state) but I believe the beta went out with a bunch of buildbots offline or failing. A related but independent issue is: should the community buildbots continue to exist? An automated test is only as good as the consequences of its failure. As it stands, the community buildbots seem to be nothing but an embarrassment. I don't mean this to slam Grig, or the Python team - I mean, literally, if there is no consequence of their failure then the only use I can see for the data the buildbots currently provide (i.e. months and months of failures) is to embarrass the Python core developers. python.org's purpose should not be to provide negative press for Python ;), so if this continues to be the case, they probably shouldn't be linked. This isn't just an issue with changes to Python breaking stuff; the pybots' configuration is apparently not well maintained, since the buildbots which say "2.5" aren't passing either, and that's not a problem with the code, it's just that the slaves are actually running against trunk@HEAD. Ultimately these tools only exist to ensure the quality of Python releases. The really critical question here is, what does it mean to have a Python release that is high-quality? There are some obvious things: it shouldn't crash, it should conform to its own documentation. Personally I think "it passes all of its own tests" and "it runs existing Python code" are important metrics too, but the most important thing I'm trying to get across here is that there should be a clear understanding of which goals the release / QA / buildbot / test process is trying to accomplish. For me, personally, it really needs to be clear when I can say "you guys screwed up and broke stuff", and when I just have to suck it up and deal with the consequences of a new version of Python in Twisted. It's definitely bad for all of us if the result is that new releases of Python just break everything. Users don't care where the responsibility lies, they just know that stuff doesn't work, so we should decide on a process which allows Twisted (and other popular projects, like Django, Plone, pytz, PyGTK, pylons, ...) to (mostly) run when new versions of Python are released. [1]: http://mail.python.org/pipermail/python-dev/2008-May/079212.html

16 57

Re: [Python-Dev] [Python-checkins] r64424 - in python/trunk:Include/object.h Lib/test/test_sys.py Misc/NEWSObjects/intobject.c Objects/longobject.c Objects/typeobject.cPython/bltinmodule.c
by Guido van Rossum 04 Jul '08

04 Jul '08

[Adding back the list.] On Tue, Jun 24, 2008 at 9:53 PM, Raymond Hettinger <python(a)rcn.com> wrote: >> While I think it's fine to have some function that reveals the binary >> representation of floats, I don't think that overlaying this on >> hex/oct/bin is worth the problems it causes. > > What problems? The patch is clean. Problems like no two people on python-dev agreeing on how exactly the feature should be implemented. Problems like whether this goes way beyond the philosophical underpinnings of bin/oct/hex. Problems like what to do about other types that might want to overload hex/oct/bin. See Kevin Jacobs' response. >> This API appears to be >> purely for educational purposes; why not implement something in pure >> Python using the struct module that reveals the lay-out of the >> floating-point value? > > This is not the internal floating point layout. It is the real value > expressed in exponential form. It is more than educational -- it is a > platform independent representation (look at Terry's reference -- it is the > usual way to precisely specify a float value and it does not depend on > atof() or vice versa). Possibly, but it is only readable by a Python expression parser. For all practical purposes "%.17g" % x works just as well. And this bypasses the question "why overload this functionality on bin/hex/oct rather than adding e.g. a new function to math or a new method to float." >> (There are also several things wrong with the specific patch, apart >> from its lack of docs; #1 is the introduction of an externaly visible >> symbol that doesn't start with _Py.) > > Will change the global symbol to _Py. I already added docs to the patch. > Did you see the one that was uploaded a few hours ago (float6.diff)? I don't care about the details of the patch until we have agreement about which form the feature should take. We don't have that agreement yet. I mentioned the flaws in the patch to point out that it was apparently a rush job. > I re-opened the discussion at your behest. [...] I'm very glad you're giving the discussion a second chance. Please give it a few days at least. My expectation is that the outcome will be not to overload bin/hex/oct but to add a custom function to math or a custom method to float, whose output can be further massaged to create the platform-independent representation you're after. (I doubt that it's worth changing pickle or marshal though, they are doing fine with their respective current approaches.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

14 58

repeated keyword arguments
by tomer filiba 02 Jul '08

02 Jul '08

the following code works on python 2.5: >>> def f(**kwargs): ... print kwargs ... >>> f(a=5,b=7,a=8) {'a': 8, 'b': 7} >>> but fails on python2.4, saying that "a" is given twice. is this a bug or a feature? -tomer

13 23

Re: [Python-Dev] urllib, multipart/form-data encoding and file uploads
by Chris AtLee 01 Jul '08

01 Jul '08

On Fri, Jun 27, 2008 at 11:40 AM, Bill Janssen <janssen(a)parc.com> wrote: >> I notice that there is some work being done on urllib / urllib2 for >> python 2.6/3.0. One thing I've always missed in urllib/urllib2 is the >> facility to encode POST data as multipart/form-data. I think it would >> also be useful to be able to stream a POST request to the remote >> server rather than having requiring the user to create the entire POST >> body in memory before starting the request. This would be extremely >> useful when writing any kind of code that does file uploads. >> >> I didn't see any recent discussion about this so I thought I'd ask >> here: do you think this would make a good addition to the new urllib >> package? > > I think it would be very helpful. I'd separate the two things, > though; you want to be able to format a set of values as > "multipart/form-data", and do various things with that resulting > "document", and you want to be able to stream a POST (or PUT) request. How about if the function that encoded the values as "multipart/form-data" was able to stream data to a POST (or PUT) request via an iterator that yielded chunks of data? def multipart_encode(params, boundary=None): """Encode ``params`` as multipart/form-data. ``params`` should be a dictionary where the keys represent parameter names, and the values are either parameter values, or file-like objects to use as the parameter value. The file-like object must support the .read(), .seek(), and .tell() methods. If ``boundary`` is set, then it as used as the MIME boundary. Otherwise a randomly generated boundary will be used. In either case, if the boundary string appears in the parameter values a ValueError will be raised. Returns an iterable object that will yield blocks of data representing the encoded parameters.""" The file objects need to support .seek() and .tell() so we can determine how large they are before including them in the output. I've been trying to come up with a good way to specify the size separately so you could use unseekable objects, but no good ideas have come to mind. Maybe it could look for a 'size' attribute or callable on the object? That seems a bit hacky... A couple helper functions would be necessary as well, one to generate random boundary strings that are guaranteed not to collide with file data, and another function to calculate the total size of the encoding to be used in the 'Content-Length' header in the main HTTP request. Then we'd need to change either urllib or httplib to support iterable objects in addition to the regular strings that it currently uses. Cheers, Chris

3 3

urllib, multipart/form-data encoding and file uploads
by Chris AtLee 01 Jul '08

01 Jul '08

Hello, I notice that there is some work being done on urllib / urllib2 for python 2.6/3.0. One thing I've always missed in urllib/urllib2 is the facility to encode POST data as multipart/form-data. I think it would also be useful to be able to stream a POST request to the remote server rather than having requiring the user to create the entire POST body in memory before starting the request. This would be extremely useful when writing any kind of code that does file uploads. I didn't see any recent discussion about this so I thought I'd ask here: do you think this would make a good addition to the new urllib package? Cheers, Chris

4 4

Cycle collection enhancement idea
by Eyal Lotem 30 Jun '08

30 Jun '08

I see why a cycle that has multiple objects with a __del__ method is a problem. Once you call __del__ on one of the objects, its no longer usable by the others, and its not clear which order is correct. My question regards the case where a cycle of objects only has 1 object which has a __del__. I think a correct strategy to collect the entire cycle is the same one used on a single object. On a single object Python uses: 1. Temporarily revive object 2. Call __del__ 3. Unrevive object (if(--refcount == 0) then we're done), otherwise it was resurrected). We can apply this to the whole cycle: 1. Temporarily revive entire cycle (each of its objects) 2. Call __del__ 3. Unrevive the objects of the entire cycle (each of its objects). Step 1 will allow __del__ to run safely. Since there is only one __del__ in the cycle, it is not dangerous that its references will disappear from "under its feet". (Some code restructuring will probably be necessary, because of assumptions that are hard-coded into slot_tp_del and subtype_dealloc). I believe this enhancement is important, because: A. When using existing code -- you do not control whether its objects have a __del__. In my experience, a majority of these cases only have a single __del__-containing object in their cycles. B. Python's exit cleanup calls __del__ in the wrong order, and Python's runtime is full of cycles (Each global is a cycle, including the class objects themselves: class->dict->function->func_globals)). These cycles very often have only 1 __del__ method. Some examples of the problem posed by B: http://www.google.com/search?q=ignored+%22%27NoneType%27+object+has+no+attr… Ugly workarounds exist even in the standard library [subprocess]: "def __del__(self, sys=sys):"). Example: import os class RunningFile(object): filename = '/tmp/running' def __init__(self): open(self.filename, 'wb') def __del__(self): os.unlink(self.filename) running_file = RunningFile() The deller object is in a cycle as described above [as well as the Deller class itself]. When Python exits, it could call deller.__del__() and then collect the cycle. But Python does the wrong thing here, and gets rid of the globals before calling __del__: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'unlink'" in <bound method RunningFile.__del__ of <__main__.RunningFile object at 0x7f9655eb92d0>> ignored I believe applying the above enhancement would solve these problems.

8 22

GC Proposal
by Adam Olsen 30 Jun '08

30 Jun '08

We need two counters: one is the total number of traceable objects (those we would inspect if we did a full collection) and a number of "pending" trace operations. Every time an object is moved into the last generation, we increase "pending" by two - once for itself and once for an older object. Once pending equals the total number of traceable objects we do a full collection (and reset "pending" to 0). This behaves well under both extremes. First, if allocating in a burst, it waits until the total number of objects has doubled before doing a collection, which works out to about 2 passes per object. Second, a long running program may allocate an object, get it into the oldest generation (adding to "pending"), but then delete it through normal means. The pending trace is effectively borrowed by the remaining older objects, which ensures they will get scanned again eventually. However, we already payed for it when the young object was allocated, so the net cost is still a constant factor. Various refinements are possible, such as only adding 1.5 to "pending", or taking off some of the 2 if a young object is deleted without getting traced. We could also tweak when in the object life-cycle "pending" is increased. The first two generations would remain as they are today. -- Adam Olsen, aka Rhamphoryncus

3 9