darn. I was hoping to get a 2.5.3 rc and final out soon. Can anyone else build the binaries?

On Tue, Aug 12, 2008 at 2:43 PM, "Martin v. Löwis" martin@v.loewis.de wrote:

...

It sounds like Wednesday August 13th will not be feasible, so we'll do beta 3 on Wednesday August 20th. I've updated both the PEP and the Google Calendar.

I'll be on vacation then, and not be able to produce Windows binaries (until September 8)

Regards, Martin _______________________________________________ python-committers mailing list python-committers@python.org http://mail.python.org/mailman/listinfo/python-committers

I am planning to offer a single file patch for 2.3 and 2.4. As far as one more 2.5 release, I don't think there's going to be many changes to the 2.5 branch between now and 2.6/3.0 final - although if there is, we'll obviously have to do another release.

I wouldn't be horribly surprised if more nasty lurking bugs remain in the C code. The google security review found a bunch, apple found some, but there's really quite a lot of code there...

On Tue, Aug 12, 2008 at 4:28 PM, "Martin v. Löwis" martin@v.loewis.de wrote:

Anthony Baxter wrote:

...

darn. I was hoping to get a 2.5.3 rc and final out soon. Can anyone else build the binaries?

For the 2.5, the challenge is to produce AMD64 and Itanium binaries, using vsextcomp (plus the usual problems of collecting all the necessary packages and build them first).

Are you also planning to produce a 2.4 security (source only) release? Will the 2.5 release be the final bug fix release?

Regards, Martin

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 12, 2008, at 3:38 AM, Martin v. Löwis wrote:

Anthony Baxter wrote:

...

I am planning to offer a single file patch for 2.3 and 2.4. As far as one more 2.5 release, I don't think there's going to be many changes to the 2.5 branch between now and 2.6/3.0 final - although if there is, we'll obviously have to do another release.

I would like to establish a tradition where, after some final bug fix release (e.g. for 2.5), further "mere" bug fixes are banned from the maintenance branch (and I did revert several bug fixes from the 2.4 branch).

I'm not sure I agree with this policy. Can you elaborate on /why/ you
want this?

I understand that we're a volunteer organization, and that our
resources are limited. I'm also wary about siphoning off those limit
resources right now for working on other than 2.6 and 3.0, but I'm not
sure that this policy really buys us much there. E.g. with this policy
you'll need a release cop to patrol commits and back out non-security
fixes right away. It's much more work to revert such changes whenever
we get around to doing the release. Seems like it could be /more/
work with this policy.

I do agree that we need to be very careful about introducing new
features, but I think non-security patches can be okay.

I had some 2.4 patches backed out because they weren't security
releases. I was okay with it at the time, but I'm uncomfortable about
imposing this as a general rule. If we have bugs, and we have someone
motivated to fix them, we should opt toward fixing them. It's
demoralizing to have one's patches backed out. Besides, we still have
downstream vendors that are maintaining and releasing Python 2.4; does
this mean they're out of luck for bug fixes or they have to roll their
own?

We're on an 18 month release schedule, which is a long time to wait,
so I'm not in favor of an arbitrary date for cutting off "mere" bug
fixes. Rather, I'd like to see a policy (but not a promise) of
supporting two releases at a time. Thus, when 2.6 is released, we
would stop support for all but critical security fixes for 2.4, but we
could (not will) still release bug fixes for 2.5. And of course,
we'll support 2.6/3.0 while 2.7/3.1 is being developed.

Having lockstep 2.x and 3.x release complicates things a bit, but
because they are lockstep, I'm thinking of them more as the same
release rather than separate ones.

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 12, 2008, at 2:44 PM, Martin v. Löwis wrote:

...

...

...

I am planning to offer a single file patch for 2.3 and 2.4. As
far as one more 2.5 release, I don't think there's going to be many
changes to the 2.5 branch between now and 2.6/3.0 final - although if there is, we'll obviously have to do another release.

...

I would like to establish a tradition where, after some final bug
fix release (e.g. for 2.5), further "mere" bug fixes are banned from the maintenance branch (and I did revert several bug fixes from the 2.4 branch).

I'm not sure I agree with this policy. Can you elaborate on /why/
you want this?

Because there won't typically be sufficient testing and release infrastructure to allow arbitrary bug fixes to be committed on the branch. The buildbots are turned off, and nobody tests the release candidate, no Windows binaries are provided - thus, chances are very high that a bug fix release for some very old branch will be *worse* than the previous release, rather than better.

Why is that qualitatively different than a security fix? All the same
conditions apply.

An alternative would be to keep all infrastructure up and running, but that is infeasible.

Or to adopt tools that help improve reliability. I'm not convinced
that the buildbots really do that. A PQM-style approach, while more
of a pain for developers because of the serialized landings, would
definitely improve things, and there's not nearly as much
infrastructure involved to keep humming for old releases. PQM isn't
perfect, but I do believe it would help.

...

I understand that we're a volunteer organization, and that our
resources are limited. I'm also wary about siphoning off those limit resources right now for working on other than 2.6 and 3.0, but I'm not sure
that this policy really buys us much there. E.g. with this policy you'll
need a release cop to patrol commits and back out non-security fixes right away.

That's not necessary. When I made 2.3.7 and 2.4.5, I went through the complete log, and posted a list of patches that I wanted to revert. This was little effort, and I'm sure it would have been even less
effort if people had known that 2.4.x is a closed branch.

I'm glad it wasn't much effort. Would you propose using technological
means to close the branch?

...

It's much more work to revert such changes whenever we get around to doing the release. Seems like it could be /more/ work with this
policy.

It wasn't really that much work - there were little non-security patches, and they were easily identifiable from the commit message.

...

I do agree that we need to be very careful about introducing new features, but I think non-security patches can be okay.

They aren't, as they don't get sufficient testing.

Again, I don't think that's qualitatively much different for security
patches. We may manually test them, inspect them, rely on vendors to
have tested them, but they don't go through the Q/A process we enforce
for our active branches.

...

It's demoralizing to have one's patches backed out. Besides, we still
have downstream vendors that are maintaining and releasing Python 2.4;
does this mean they're out of luck for bug fixes or they have to roll
their own?

I've talked to the downstream vendors, and they really want security patches for a long time, above all. They are fine with maintaining
their own patches (which they do, anyway).

Would a policy of security-patches-only have any effect on vendors
sharing fixes with us? By that I mean, if 2.4 were open to non- security patches, would they be more or less willing to feed them
upstream, where we could, if someone were motivated, port them forward?

...

We're on an 18 month release schedule, which is a long time to
wait, so I'm not in favor of an arbitrary date for cutting off "mere" bug
fixes. Rather, I'd like to see a policy (but not a promise) of supporting
two releases at a time.

I think this requires more resources than we have - especially with your way of counting:

...

Thus, when 2.6 is released, we would stop support for all but critical security fixes for 2.4, but we could (not will) still release bug fixes for 2.5. And of course, we'll support
2.6/3.0 while 2.7/3.1 is being developed.

So that's *three* branches that we need to maintain, right: 2.5, 2.6, and 3.0. Once 3.1 is released, I supposed it's even *four* branches: 2.6, 2.7, 3.0, and 3.1. That means that every change must be committed/merged four times, you need to run the test suite four
times, and so on. Depending on the nature of the bug fix, you also need to keep the specifics of the four branches in mind.

I don't think our committers will accept such a work load - which
means that some patches don't get backported (arbitrarily, depending on how relevant the committer views that policy).

Let me emphasize that I'm not suggesting our committers do this. I'm
suggesting that if a specific committer is motivated to fix a non- security bug in an older release, they would have to accept this
responsibility. Maybe it'll never happen because no one really cares
enough. But a policy against it would /prevent/ it even if there was
motivation to do it.

...

Having lockstep 2.x and 3.x release complicates things a bit, but because they are lockstep, I'm thinking of them more as the same
release rather than separate ones.

I think this is an illusion. When did you last commit something to the trunk, and forward-ported it to the 3.0 branch? When did you last run "svnmerge avail"? Porting patches between 2.6 and 3.0 is anything but trivial.

I'll concede that it's very difficult.

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 13, 2008, at 6:41 PM, Antoine Pitrou wrote:

Le mercredi 13 août 2008 à 18:33 -0400, Barry Warsaw a écrit :

...

Or to adopt tools that help improve reliability. I'm not convinced that the buildbots really do that. A PQM-style approach, while more of a pain for developers because of the serialized landings, would definitely improve things, and there's not nearly as much infrastructure involved to keep humming for old releases. PQM isn't perfect, but I do believe it would help.

What is a "PQM-style approach"?

PQM = Patch Queue Manager

Basically, it's a robot that controls commits to the trunk. Nothing
lands in the trunk without getting through PQM first. PQM serializes
changesets so that they must apply cleanly with no conflicts, and pass
the entire test suite. There could be other conditions, e.g. that it
lints cleanly, has no whitespace issues, etc.

If any of the set of conditions fail, the changeset does not land.
This means that the trunk is always in a releasable state, and we
avoid the problems I run into all the time now, where we have red
buildbots on or near release date. I would dearly love to be able to
spin a release at any time and have a high degree of confidence that
what I'm releasing is stable.

There's a specific implementation of PQM based on the Bazaar revision
control system, available here: https://edge.launchpad.net/pqm

PQM is not perfect, nor is it a perfect fit for us. For example, we
have buildbots that run on multiple platforms, while PQM runs on a
single platform. So a vanilla PQM could still miss changes that break
only on a specific operating system. It also doesn't help at all for
bugs not covered by the test suite (well, buildbots don't help there
either ;).

PQM also introduces delays on trunk landing because it serializes
commits. So when things get backed up, it might take a while for your
branch to land on the trunk.

PQM wouldn't replace the buildbots, but it would greatly improve the
quality of the development branches, IMO. The buildbots would still
be useful to ensure cross-platform quality.

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 13, 2008, at 7:52 PM, Bill Janssen wrote:

...

PQM serializes changesets so that they must apply cleanly with no conflicts, and pass the entire test suite.

What platform would it run the test suite on? Presumably the same one I tested on before I submitted the patch :-).

I think we'd just have to pick one. It would probably be a *nix based
system.

I think this works if you're a Linux development shop, but perhaps not as well for Python.

It would still solve a problem we have today, which is that the
release branch is very often broken when the time comes to cut a
release. We've had to delay several releases because of red buildbots
or failing tests across multiple platforms. Even having the branch
always releasable on <pick one> Linux would be a big improvement.

• -Barry

...

What platform would it run the test suite on? Presumably the same one I tested on before I submitted the patch :-).

I think we'd just have to pick one. It would probably be a *nix based
system.

So everyone who wanted to work on the system would effectively have to go out and get one of those. To my mind, that's a non-starter.

Bill

On Wed, Aug 13, 2008 at 6:12 PM, Christian Heimes christian@cheimes.de wrote:

Barry Warsaw wrote:

...

PQM = Patch Queue Manager

Basically, it's a robot that controls commits to the trunk. Nothing lands in the trunk without getting through PQM first. PQM serializes changesets so that they must apply cleanly with no conflicts, and pass the entire test suite. There could be other conditions, e.g. that it lints cleanly, has no whitespace issues, etc.

Personally I'm totally against any kind of tool like PQM for general development. Issues due erroneous check-ins are a social problem. I strongly believe that social problems can't be solved by a system like PQM. PQM may work for companies or projects with a large developer group but not for Python. I fear it'd cause more problems than it's worth. There are valid reasons for checking in failing unit tests. For example a developer spots a problem but isn't able to fix on his own. Any fancy system that delays or prohibits check-ins is going to slow us down.

In my opinion a system like PQM should only be used when a RC or final release is immanent. I can picture the usefulness of PQM during the last few weeks before a release.

I'd rather see the man power put into better testing facilities than into a tool like PQM. If you are worried about the stability of the trunk I'd rather suggest a change of our code of conduct. For example every change of code, which isn't just a minor change, must be applied to a new branch and reviewed by a second developer before it's applied to the trunk. I think development inside branches and peer reviewing yield better results than a machine that rules over developers.

Christian, who still thinks (hopes) that the human mind outperforms machines when it comes down to important and complex decisions.

[Hey Christian, welcome back! (It seems we hadn't heard much from you for a while...)]

I don't have experience with PQM or something like it, but I suspect it doesn't scale, and the buildbots are a better approach, because they handle multiple platforms.

I do think that better tools can help us some -- while you can't solve every social problem with tools, the reverse isn't true either -- sometimes it *is* possible to solve (or at least alleviate) social problems with tools.

As long as we're touting tools or processes that we have experience with, Google uses a combination of tools. One tool is similar to the buildbots, running tests *after* stuff has been checked in. A feature that buildbot is missing is that it tries to figure which checkin is responsible for a particular failure, and mails both the author of that change and the owner of the code (if different).

Another tool that helps code quality tremendously is mandatory peer reviews of all code before checkin. Call me biased because I wrote the first version of the tool that most Googlers today use for peer reviews (Collin Winter now runs that show) and also open-sourced a similar tool (Rietveld, http://codereview.appspot.com) that works with Subversion. But I didn't set the policy of mandatory pre-checkin peer reviews -- Google had that policy for years before, and other tools had already been written.

I particularly like the peer review policy because it is actually largely a *social* solution: code review is a very social process, the tools are secondary. By peer-reviewing all code before it goes in, not only do you catch many problems in an earlier stage than otherwise possible, but it also ensures that there are always at least two developers who know and understand the changes (or new code -- everything gets reviewed, even doc changes and changes to build files). Plus, it improves relationships between developers: if yesterday I found a bug in your code during peer review, today I may be more susceptible to you pointing out bugs in *my* code. This is a lot better than finding bugs during debugging sessions: once you've spent an hour tracking down a nasty bug you're much more likely to have a low opinion of whoever checked in the buggy code, and you're less likely to trust their judgment of your own code.

Anyway, if we're going to change policies around submitting code, I would much rather see peer review become a habit than adopt a tool like PQM.

Hey Barry,

First, let me thank you for taking the energy to express many interesting points in that discussion.

...

I don't have experience with PQM or something like it, but I suspect it doesn't scale, and the buildbots are a better approach, because they handle multiple platforms.

(...)

• • PQM: right, doesn't scale across multiple platforms, but still valuable.

Guards against broken mainline affecting everybody. E.g. would have caught the multiprocessing bug that delayed an earlier release affecting at least Linux and OS X.

I'm not sure I get what's the scaling issue here. It's likely that you guys have some experience that I lack, and thus are talking about an obvious issue that I'm not aware of.

We're actually adopting a PQM-based approach *precisely* because we have to maintain a code base in multiple versions of the same platform, and thus can't be sure that the tests will continue working just by running the suite in our own environments. In this case PQM will help precisely because running tests locally "doesn't scale", which is why I'm a bit surprised by the straightforward agreement on the point above.

What exactly is the "doesnt' scale" issue in this case?

Hey Martin,

...

What exactly is the "doesnt' scale" issue in this case?

I don't actually know PQM, but from what I hear, it appears that PQM can't run tests for a single patch on all of Solaris, Linux, Windows, and OS X before accepting it. Instead, PQM will run the tests on the same machine it runs on itself, and only on that machine.

We actually have PQM setup to work in multiple machines for the same commit, since we have to ensure that the same source code passes all tests in a number of platform versions. I don't personally know in which platforms PQM runs.

In any case, I'm not suggesting Python should use PQM-the-implementation. Maybe it's too complex for what is necessary, or too simple, or not portable enough, or whatever. At this point, I'm only trying to point out that if you want stability on platforms that differ from the local one in which a developer run the test suite on, you must block the merges that will break the other platforms.

Even though a bit obvious, this idea is still not very popular nowadays because for a long time, not merging broken code meant not keeping evolving source code under revision control. This doesn't have to be true now. Developers may evolve their code in a broken way for a long time before integrating in a main line of development, and consequently without creating breakage for others, and blocking releases.

As long as we're touting tools or processes that we have experience with, Google uses a combination of tools. One tool is similar to the buildbots, running tests *after* stuff has been checked in. A feature that buildbot is missing is that it tries to figure which checkin is responsible for a particular failure, and mails both the author of that change and the owner of the code (if different).

Buildbot does have a blame list. Mailing the developer failed so far because we didn't have email addresses of each developer. Now, with the python-committers list, Brett collected them, so such mailing would be possible.

I'm not sure about mailing the owner of the code: we don't typically have owners of code, right? Would you consider it useful to support the case were we do have an owner? Such an ownership database would need to be maintained, as well, and typically, when we do have owners, the owner is also the committer.

Another tool that helps code quality tremendously is mandatory peer reviews of all code before checkin. Call me biased because I wrote the first version of the tool that most Googlers today use for peer reviews (Collin Winter now runs that show) and also open-sourced a similar tool (Rietveld, http://codereview.appspot.com) that works with Subversion. But I didn't set the policy of mandatory pre-checkin peer reviews -- Google had that policy for years before, and other tools had already been written.

So would you enforce that policy with tools, too? If so, what would the workflow look like?

Anyway, if we're going to change policies around submitting code, I would much rather see peer review become a habit than adopt a tool like PQM.

The part where I'm skeptical about such a policy is that there might be a shortage of reviewers. What if a patch on Rietvield doesn't find a reviewer for a month or so? Many patches in the tracker sit there for years without any committer reviewing them.

Regards, Martin

Le jeudi 14 août 2008 à 08:54 +0200, "Martin v. Löwis" a écrit :

The part where I'm skeptical about such a policy is that there might be a shortage of reviewers. What if a patch on Rietvield doesn't find a reviewer for a month or so? Many patches in the tracker sit there for years without any committer reviewing them.

There's also the problem that some patches can only be understood by one person, the one who wrote it, and it's somehow still fine because he/she is the de facto maintainer for that part of code (e.g. ctypes, bsddb...).

("not finding a reviewer for a month or so" is actually an optimistic statement; from my short experience with Twisted it can be much longer than that :-))

That said, I appreciate the advantages of code reviews. But they do take a lot of time, and can create manpower problems (by lowering throughput *and* increasing latencies).

It would be a change in culture for us, that's for sure. The question becomes whether the drop in patch throughput is justified by an increase of patch quality and stability in the code?

Well, let's take the multiprocessing example. The question is:

• would an a priori review have been able to uncover the most subtle bugs?
• would someone have done that review at all, and how long would it have

taken before it had been done?

• if we had delayed the inclusion of multiprocessing in the mainline,

doesn't it mean it would have got almost no testing since people are unlikely to test specific branches rather than the "official trunk"?

• isn't the inclusion of multiprocessing itself, even if subtle bugs

remain, an increase of quality since it gives people a standard and rather good package for doing parallel stuff, rather than baking their own defective ad-hoc solutions?

The thing I want to point out in the latter item is that measuring quality solely by the number of bugs or the stability of a bunch of buildbots is wrong (although of course fixing bugs and having green buildbots *is* important). Sometimes committing an imperfect patch can be better than committing nothing at all.

Regards

Antoine.

Barry Warsaw wrote:

I think this is a case where PQM might have helped. Assuming the build/test would uncover these subtle bugs, the multiprocessing code would not have landed. You would then probably publish a branch with d t those (failing) changes and rally the help you needed to fix those problems. Then you'd try to land it again.

The workflow would have likely been very similar to what happened for this code, except that it would be happening on a branch, not on the mainline. Maybe no one would have been motivated enough to get them working if they weren't breaking mainline. The tradeoff is instability in the mainline and uncertainty as to whether the mainline is of high enough quality to release.

I suggest that we should use branches to a greater extend. New features or updates of existing code should happen on a branch. A branch must not be merged until it's tested on all major platforms (Linux i386, Linux AMD64, Mac OS X and Win32) and peer reviewed by another developer.

During my time as a Zope and Plone developer and at various XP sprints I've utilized the branch development and peer reviewing workflow with great success. I assume the majority of Python developers don't do branches because it's an expensive and annoying operation with svn. Well branching isn't so annoying but merging and keeping a branch up to date definitely is. Once we have a VCS with cheap branching and easy merging we should switch to branched development.

Christian

Christian Heimes schrieb:

I suggest that we should use branches to a greater extend. New features or updates of existing code should happen on a branch.

+1

A branch must not be merged until it's tested on all major platforms (Linux i386, Linux AMD64, Mac OS X and Win32) and peer reviewed by another developer.

+1 for the major platforms, -0 for the peer review.

During my time as a Zope and Plone developer and at various XP sprints I've utilized the branch development and peer reviewing workflow with great success. I assume the majority of Python developers don't do branches because it's an expensive and annoying operation with svn.

Actually I don't find branching expensive - why so you think so? Annoying - hm, no, I'm not sure. The only additional work that should be done is to merge the branch when (if!) it's ready, delete the branch when it is not needed any more, and remove the local branch checkout(s).

Well branching isn't so annoying but merging and keeping a branch up to date definitely is. Once we have a VCS with cheap branching and easy merging we should switch to branched development.

Well, svnmerge makes keeping a branch up to date pretty painless imo.

I did the pep3118 implementation for ctypes in a separate branch, I did upgrade the years-old libffi in a separate branch, maybe even more.

This worked well for me, and the big advantage was that I coud trigger builds for these branches on the buildbots on platforms where I have no other access to.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 14, 2008, at 2:54 AM, Martin v. Löwis wrote:

...

As long as we're touting tools or processes that we have experience with, Google uses a combination of tools. One tool is similar to the buildbots, running tests *after* stuff has been checked in. A feature that buildbot is missing is that it tries to figure which checkin is responsible for a particular failure, and mails both the author of that change and the owner of the code (if different).

Buildbot does have a blame list. Mailing the developer failed so far because we didn't have email addresses of each developer. Now, with the python-committers list, Brett collected them, so such mailing would be possible.

I'm not sure about mailing the owner of the code: we don't typically have owners of code, right? Would you consider it useful to support the case were we do have an owner? Such an ownership database would need to be maintained, as well, and typically, when we do have owners, the owner is also the committer.

I'm not in favor of code owners, but commit owners might be
worthwhile. If we have the email addresses now, and can easily enable
it, then I think we should have the buildbots email both this mailing
list and the owner of the commit when things break.

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 13, 2008, at 9:12 PM, Christian Heimes wrote:

Barry Warsaw wrote:

...

PQM = Patch Queue Manager Basically, it's a robot that controls commits to the trunk.
Nothing lands in the trunk without getting through PQM first. PQM
serializes changesets so that they must apply cleanly with no
conflicts, and pass the entire test suite. There could be other
conditions, e.g. that it lints cleanly, has no whitespace issues,
etc.

Personally I'm totally against any kind of tool like PQM for general
development. Issues due erroneous check-ins are a social problem. I
strongly believe that social problems can't be solved by a system
like PQM.

Unfortunately, they're not being solved without PQM either! Really,
we've had to delay releases several times because the branches were
broken across multiple operating systems. Pleading on the mailing
lists doesn't help. Hanging out on irc doesn't help. Having
Benjamin, Georg, and others kicking ass on #python-dev the day of a
release, is great, but it's also asking a lot of them.

PQM may work for companies or projects with a large developer group
but not for Python. I fear it'd cause more problems than it's worth. There are valid
reasons for checking in failing unit tests. For example a developer
spots a problem but isn't able to fix on his own. Any fancy system
that delays or prohibits check-ins is going to slow us down.

That's what branches are for. I really strongly feel that the
mainlines (by which I mean the branches we cut releases from) should
always be in a releasable state. We should never be committing broken
tests to these mainlines. If you spot a problem you can't fix, create
a branch and commit the broken test there, and ask for help with that
branch. The mainline isn't (IMHO) the place for that.

You're right that it will slow us down, but only on the mainline.
That's a good thing, especially if it buys you high quality.

In my opinion a system like PQM should only be used when a RC or
final release is immanent. I can picture the usefulness of PQM
during the last few weeks before a release.

We should be using the same quality assurance early in the cycle that
we do late in the cycle. Realistically, we're never going to switch
to something different when we get to RC.

I'd rather see the man power put into better testing facilities than
into a tool like PQM. If you are worried about the stability of the
trunk I'd rather suggest a change of our code of conduct. For
example every change of code, which isn't just a minor change, must
be applied to a new branch and reviewed by a second developer before
it's applied to the trunk. I think development inside branches and
peer reviewing yield better results than a machine that rules over
developers.

These are good policies to adopt. I know of many projects that
require one or two positive code reviews for branches to land in the
mainline. Code reviews and PQM augment each other though, they don't
replace each other. We're all human and code reviews will never be
perfect. Some changes have non-local or unexpected effects that only
the test suite will catch. Maybe the test pass on your machine
because of something in your environment that breaks in the PQM
"ideal" environment.

Christian, who still thinks (hopes) that the human mind outperforms
machines when it comes down to important and complex decisions.

It's not us vs. them. I want the machines to do the crappy grunt work
so us humans can do what we do best, being creative and having
fun :). Begging people to fix broken buildbots on release day is
neither.

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 14, 2008, at 10:33 AM, Christian Heimes wrote:

By the way the guys are totally awesome, dude. :)

I agree wholeheartedly!

...

That's what branches are for. I really strongly feel that the
mainlines (by which I mean the branches we cut releases from)
should always be in a releasable state. We should never be
committing broken tests to these mainlines. If you spot a problem
you can't fix, create a branch and commit the broken test there,
and ask for help with that branch. The mainline isn't (IMHO) the
place for that. You're right that it will slow us down, but only on the mainline.
That's a good thing, especially if it buys you high quality.

Sticking to our own rules would also buy us quality ... Let's not
add new features to our code base during the beta phase, please.
Although the addition of multiprocessing had some merit, we
shouldn't to the same mistake twice.

That wouldn't have helped. multiprocessing was added during the alpha
phase.

Perhaps we could adopt a release plan similar to Ubuntu. They have
releases with cool, new and bleeding edge stuff followed by a
release that focuses on stability and long term support. Python 2.6
and especially 3.0 are releases with new features. What do you think
about focusing on stability and long time support for 2.7 and 3.1?
2.7 might be the last version of the 2.x series and we sure gonna
have to fix lots of issues in the 3.x series until it's matured.

If we did this, I think it should be less than 18 months between
releases. But I also fear that there will be too much pressure to add
new features anyway.

I remember at some distant Python conference, Guido asked the
audience, how many people feel Python is changing too fast, and then
how many people feel it's missing an important feature. IIRC, the
show of hands was about equal.

all-features-except-mine-are-unimportant-ly y'rs,

• -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 14, 2008, at 6:16 PM, Jesse Noller wrote:

Yup - it went in during alpha, and I underestimated the amount of
work, which won't happen again.

Stunning revelation - getting everything right cross platform is hard.

Note, mp was not the only late-stage addition, there were other core
language (non package) things in flux as well

Let me take this opportunity to explicitly say that I'm not /blaming/
anybody for this. I consider the problems we had with the module an
indication of a systemic, team-wide deficiency. I greatly appreciate
you (and others) working so hard to land it and make it work.

• -Barry

Barry Warsaw wrote:

Let me take this opportunity to explicitly say that I'm not /blaming/ anybody for this. I consider the problems we had with the module an indication of a systemic, team-wide deficiency. I greatly appreciate you (and others) working so hard to land it and make it work.

Hey! Somebody is reading my mind here ... By any chance are you telepathic? *g*

Le jeudi 14 août 2008 à 16:33 +0200, Christian Heimes a écrit :

Perhaps we could adopt a release plan similar to Ubuntu. They have releases with cool, new and bleeding edge stuff followed by a release that focuses on stability and long term support.

Ubuntu's sophisticated release plan is certainly justified by its business model, and the desire to both appeal to the open source people and the corporate people without creating two different distributions.

I don't think Python has the same business requirements (neither does it have marketing and commercial teams), and having differentiated releases sounds like unwarranted complication.

Just my two non-Canonical cents of course!

What would this professionalisation get us that we don't have now? As far as I can see, the biggest hole at the moment (as always) is with people to trawl the tracker and triage bug reports and patches.

On Fri, Aug 15, 2008 at 8:35 AM, Christian Heimes lists@cheimes.de wrote:

Antoine Pitrou wrote:

...

Ubuntu's sophisticated release plan is certainly justified by its business model, and the desire to both appeal to the open source people and the corporate people without creating two different distributions.

I don't think Python has the same business requirements (neither does it have marketing and commercial teams), and having differentiated releases sounds like unwarranted complication.

IMHO the PSF and Python core development crew doesn't have a business plan at all. By business plan I'm referring to making money with Python directly. As far as I know most core developers are working on Python beause it's fun. Some developers like Guido are paid by their employers to work on Python as part of their job.

Just a crazy idea ... Maybe it's time to make the next step toward professionalizing Python. Python is more and more becoming important for companies. They have to rely upon a stable and solid Python interpreter. Perhaps some companies are willing to pay the PSF. In return the PSF could hire some developer to work on Python full time. A couple of months ago one well known core developer expressed his interest in a paid job. A crew of three to four full time developers could make a huge difference.

...

Just my two non-Canonical cents of course!

Nice pun :)

Christian _______________________________________________ python-committers mailing list python-committers@python.org http://mail.python.org/mailman/listinfo/python-committers

Anthony Baxter wrote:

What would this professionalisation get us that we don't have now? As far as I can see, the biggest hole at the moment (as always) is with people to trawl the tracker and triage bug reports and patches.

It would get us one or some developers who can work 8+ hours a day, five days a week. A paid developer could do all crummy jobs like closing bugs. A paid release manager could dedicate more time to take care of things. I strongly believe that two full time developers can make a difference by providing a better infrastructure for the rest of us.

I blindly assume that non of us is going to say "Give me bucks or I won't code any more". But our economy is going through a recession, everything is getting more expensive by the day. A constant income would allow some guys to code without them worrying about money. For example we have several talented students in our midst. Let's tap into their potential by hiring them. They could use their talent to our benefits instead of earning money through internships or side jobs.

Christian

PS: No, I'm not trying to get me a paid job here. I've already got a great and well paid one. *g*

On Wed, Aug 13, 2008 at 4:03 PM, Barry Warsaw barry@python.org wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 13, 2008, at 6:41 PM, Antoine Pitrou wrote:

...

Le mercredi 13 août 2008 à 18:33 -0400, Barry Warsaw a écrit :

...

Or to adopt tools that help improve reliability. I'm not convinced that the buildbots really do that. A PQM-style approach, while more of a pain for developers because of the serialized landings, would definitely improve things, and there's not nearly as much infrastructure involved to keep humming for old releases. PQM isn't perfect, but I do believe it would help.

What is a "PQM-style approach"?

PQM = Patch Queue Manager

Basically, it's a robot that controls commits to the trunk. Nothing lands in the trunk without getting through PQM first. PQM serializes changesets so that they must apply cleanly with no conflicts, and pass the entire test suite. There could be other conditions, e.g. that it lints cleanly, has no whitespace issues, etc.

Ah, OK.

If any of the set of conditions fail, the changeset does not land. This means that the trunk is always in a releasable state, and we avoid the problems I run into all the time now, where we have red buildbots on or near release date. I would dearly love to be able to spin a release at any time and have a high degree of confidence that what I'm releasing is stable.

Well, it would definitely catch those changes that manage to break ALL the buildbots, but those that only break on the platforms that the developer is not on might just cause pain.

There's a specific implementation of PQM based on the Bazaar revision control system, available here: https://edge.launchpad.net/pqm

Why am I not surprised that bzr has support for something you are suggesting? =)

PQM is not perfect, nor is it a perfect fit for us. For example, we have buildbots that run on multiple platforms, while PQM runs on a single platform. So a vanilla PQM could still miss changes that break only on a specific operating system. It also doesn't help at all for bugs not covered by the test suite (well, buildbots don't help there either ;).

PQM also introduces delays on trunk landing because it serializes commits. So when things get backed up, it might take a while for your branch to land on the trunk.

There is also the shift in development style to larger patch sets (assuming your version control system supports patch sets) instead of a lot of incremental commits to the trunk since you might end up with a cascading failure of your commits if you were dependent on an earlier one fails and thus all your subsequent checkins will be rejected.

PQM wouldn't replace the buildbots, but it would greatly improve the quality of the development branches, IMO. The buildbots would still be useful to ensure cross-platform quality.

Well, another option is to flesh out make check such that it runs more sanity checks and makes sure the entire test suite is run and passed before a commit is done, basically doing what you are suggesting, but on a local level.

-Brett

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Aug 13, 2008, at 10:49 PM, Brett Cannon wrote:

...

If any of the set of conditions fail, the changeset does not land.
This means that the trunk is always in a releasable state, and we avoid
the problems I run into all the time now, where we have red buildbots
on or near release date. I would dearly love to be able to spin a release at
any time and have a high degree of confidence that what I'm releasing is
stable.

Well, it would definitely catch those changes that manage to break ALL the buildbots, but those that only break on the platforms that the developer is not on might just cause pain.

That's a downside of PQM, but still, someone's gotta fix the pain.
Ideally, it would be the developer who committed the broken change.
The problem of not having access to all the platforms, or not being
able to debug problems on those platforms is a different issue.

...

There's a specific implementation of PQM based on the Bazaar revision control system, available here: https://edge.launchpad.net/pqm

Why am I not surprised that bzr has support for something you are
suggesting? =)

Hey, a good tool is a good tool. :) But really, I think the process
is the important thing, then find the tools that make that process easy.

...

PQM is not perfect, nor is it a perfect fit for us. For example,
we have buildbots that run on multiple platforms, while PQM runs on a single platform. So a vanilla PQM could still miss changes that break
only on a specific operating system. It also doesn't help at all for bugs
not covered by the test suite (well, buildbots don't help there either ;).

PQM also introduces delays on trunk landing because it serializes
commits. So when things get backed up, it might take a while for your branch
to land on the trunk.

There is also the shift in development style to larger patch sets (assuming your version control system supports patch sets) instead of a lot of incremental commits to the trunk since you might end up with a cascading failure of your commits if you were dependent on an earlier one fails and thus all your subsequent checkins will be rejected.

Yep. Figure out what you want, then find the tools that fit. :)

...

PQM wouldn't replace the buildbots, but it would greatly improve
the quality of the development branches, IMO. The buildbots would still be
useful to ensure cross-platform quality.

Well, another option is to flesh out make check such that it runs more sanity checks and makes sure the entire test suite is run and passed before a commit is done, basically doing what you are suggesting, but on a local level.

Hook that into svn commit so that you /have/ to run it, and maybe you
can convince me. :)

• -Barry

...

Because there won't typically be sufficient testing and release infrastructure to allow arbitrary bug fixes to be committed on the branch. The buildbots are turned off, and nobody tests the release candidate, no Windows binaries are provided - thus, chances are very high that a bug fix release for some very old branch will be *worse* than the previous release, rather than better.

Why is that qualitatively different than a security fix? All the same conditions apply.

No. The problem being fixed is completely different. For a security fix, it is typically fairly obvious what the bug being fixed is (in particular, if you look at the recent ones dealing with overflows): the interpreter crashes without the patch, and stops crashing (but raises an exception instead) with the patch.

For regular bug fixes, it is much more difficult to see whether the behavior being changed was a bug. They typically "merely" change the behavior A to behavior B, along with a claim that behavior A is a bug, and behavior B is correct. Even if that is true, there is still a chance that applications relied on behavior A, and break. OTOH, for an interpreter crash, it is highly unlikely that existing applications rely on the crash.

For example, in the 2.4 branch, among the patches I rolled back, was r53001. This adds string.strip to the lookup of logging handlers. It might be "better" to do that (and perhaps even correspond with the documentation), but still, it might break applications which had leading or trailing spaces in their handlers names.

...

That's not necessary. When I made 2.3.7 and 2.4.5, I went through the complete log, and posted a list of patches that I wanted to revert. This was little effort, and I'm sure it would have been even less effort if people had known that 2.4.x is a closed branch.

I'm glad it wasn't much effort. Would you propose using technological means to close the branch?

They are still open for security patches (well, 2.4 is; under my proposed policy, 2.3 isn't anymore). If people think it's desirable, we could rename the branch, or we could enforce a certain keyword (e.g. "security") in the commit messages.

Again, I don't think that's qualitatively much different for security patches. We may manually test them, inspect them, rely on vendors to have tested them, but they don't go through the Q/A process we enforce for our active branches.

Due to the reliance on inspection, it is *particularly* important that there are only few of them, and that those are all local.

Would a policy of security-patches-only have any effect on vendors sharing fixes with us? By that I mean, if 2.4 were open to non-security patches, would they be more or less willing to feed them upstream, where we could, if someone were motivated, port them forward?

I do think that vendors will continue to provide patches. They want to get rid of them to reduce their overhead, eventually, and it doesn't really matter that much that they get rid of them for the current branch (as they have done all the hard work there already). Efforts grow when you need to forward-port it, in which case you do want to contribute it upstream (in the hope of at least partially offloading the effort to a regular Python contributor).

Let me emphasize that I'm not suggesting our committers do this. I'm suggesting that if a specific committer is motivated to fix a non-security bug in an older release, they would have to accept this responsibility. Maybe it'll never happen because no one really cares enough. But a policy against it would /prevent/ it even if there was motivation to do it.

I don't like the arbitrariness that this will produce.

...

I think this is an illusion. When did you last commit something to the trunk, and forward-ported it to the 3.0 branch? When did you last run "svnmerge avail"? Porting patches between 2.6 and 3.0 is anything but trivial.

I'll concede that it's very difficult.

Indeed. I just added s* to both 2.6 and 3.0, and it took me two days to port it from 2.6 to 3.0 (just because 3.0 was using the buffer interface in so many more places).

Regards, Martin

That's true of a certain class of bugs, probably mostly in the C code. I think potential security bugs in Python code will be closer to "regular" bug fixes.

While that may be true, I think that are much more infrequent, because many attack paths (such as memory overwrites leading to remote code execution) just don't exist. In many cases, it's clear that it is the application that uses the library in an insecure manner, so many security fixes involving Python code will be in the applications, not the library.

In fact, I only recall a single security-related bug fix of Python code, advisory PSF-2005-001: XML-RPC would allow to invoke arbitrary Python code. In fixing this, valid uses might have been broken, so the patch added the allow_dotted_names attribute for such applications. Clearly, quite some time went into designing a fix for this problem, much more than for a regular bug fix. This is comparatively ok, since security fixes for Python library code are so rare.

I was thinking about preventing commits on the branch. Most security fixes of the type you describe come in through the psrt, and they may even be embargoed. For a closed branch, you'd open it for the security patches when the embargo is lifted, make the commits, then close it again. That would at least be a very strong clue that the branch is closed :).

That would be fairly easy to do. If there is consensus what branches are closed, I can come up with a pre-commit hook.

Regards, Martin

Martin v. Löwis wrote:

If you feel the urgency of doing something now, how about *only* providing patches for 2.5 at the moment? ActiveState might integrate them into ActivePython; system vendors can also pick them up (if they haven't already).

I *could* integrate patches into an ActivePython release. But I've found that there is value in having ActivePython using (mostly) the exact same sources as a python.org release. Depends on the patches, I guess.

Trent

-On [20080812 08:28], "Martin v. Löwis" (martin@v.loewis.de) wrote:

For the 2.5, the challenge is to produce AMD64 and Itanium binaries, using vsextcomp (plus the usual problems of collecting all the necessary packages and build them first).

I have a Windows x64 box at home. Which Visual Studio was now needed, 2008?

If you want Martin, we can go over making sure I got a build environment set up as well?

I'll put my hand up for doing the Windows build as well (the x64 buildbot has all the necessary bits and pieces installed). I know some HP people that I could rope in to install the resulting IA64 build and run rt.bat.

We're not shipping PGO builds for 2.5.x are we?

Martin, not sure if you've had a chance to do anything with the VeriSign certificates? (Ambiguous question mark intended ;-)

    Trent.

-----Original Message----- From: python-committers-bounces@python.org [mailto:python- committers-bounces@python.org] On Behalf Of "Martin v. Löwis" Sent: 12 August 2008 08:43 To: Jeroen Ruigrok van der Werven Cc: python-committers@python.org Subject: Re: [python-committers] [Python-Dev] next beta

Jeroen Ruigrok van der Werven wrote:

...

-On [20080812 08:28], "Martin v. Löwis" (martin@v.loewis.de)

wrote:

...

...

For the 2.5, the challenge is to produce AMD64 and Itanium

binaries,

...

...

using vsextcomp (plus the usual problems of collecting all the necessary packages and build them first).

I have a Windows x64 box at home. Which Visual Studio was now

needed, 2008?

2003, along with a platform SDK, and vsextcomp, and we would need both AMD64 and IA64 binaries (although it's probably ok if the Itanium binaries aren't tested as long as you can verify that they are IA64 binaries).

...

If you want Martin, we can go over making sure I got a build

environment set

...

up as well?

I'm leaving Sunday - not sure whether that's enough time to set things up.

Regards, Martin _______________________________________________ python-committers mailing list python-committers@python.org http://mail.python.org/mailman/listinfo/python-committers

...

I'll put my hand up for doing the Windows build as well (the x64 buildbot has all the necessary bits and pieces installed). I know some HP people that I could rope in to install the resulting IA64 build and run rt.bat.

Notice that we both look for somebody to build the next 2.6/3.0 betas (see the subject), and the 2.5.3 rc and final release (assuming Anthony still has that plan).

I'd like to offer my Win2k8 x64 colo box for building Python installers. It's a Hyper-V instance with four cores, 4GB of RAM and plenty of disk space, and has just about every Microsoft-oriented development tool installed produced this decade (inc. Itanium compilers).

So, ideally, any committer that's interested in being able to produce the official Python binaries can drop me a note and I'll give them remote access. That way we've got a single 'build server' for producing releases, as well as multiple people with the capacity to perform such builds.

I'm probably biased because it's a freakin' expensive colo and I'm severely under-utilising the box (host is an 8-core 8GB monster), but I'd argue that a nominated build server is more desirable for official releases than someone's laptop.

Thoughts?

(Even if we don't go down this path, if other developers would like remote access to the box just drop me an e-mail. I specifically acquired it to 'donate' to Python-oriented activities; no-one else has access to it.)

    Trent.