Mailman 3 August 2005 - Python-Dev

Fwd: PEP: Migrating the Python CVS to Subversion
by Guido van Rossum 15 Aug '05

15 Aug '05

Here's another POV. (Why does evereybody keep emailing me personally?) --Guido van Rossum (home page: http://www.python.org/~guido/) ---------- Forwarded message ---------- From: Daniel Berlin <dberlin(a)dberlin.org> Date: Aug 13, 2005 7:33 PM Subject: Re: [Python-Dev] PEP: Migrating the Python CVS to Subversion To: gvanrossum(a)gmail.com (Sorry for the lack of proper References: headers, this is a reply to the email archive). It's been a couple years since i've been in the world of python-dev, but apparently i'm rejoining the mailing list at just the right time. Take all of this for what it is worth: I'm currently responsible for GCC's bugzilla, wiki, in addition to maintaining several optimization areas of the compiler :P. In addition, i'm responsible for pushing GCC (my main love and world :P) towards Subversion. I should note my bias at this point. I now have full commit access to Subversion. However, I've also submitted patches to monotone, etc. We had a long thread about the various alternatives (arch, bzr, etc), and besides "freeness" constraints on what we can run on gcc.gnu.org as an FSF project, it wouldn't have mattered anyway. This has been in the planning for about a year now (mainly waiting for new hardware). Originally, we were hoping to move GCC to monotone, but it didn't mature fast enough (it's way too slow), and we couldn't make it centralized enough for our tastes (more later). The rest of the free tools other than subversion (arch, monotone, git, darcs, etc) simply couldn't handle our repository with reasonable speed/hardware. GCC has project history dating back to 1987. It's a 4 gig CVS repo containing > 1000 tags, and > 300 branches. The changelog alone has 30k trunk revisions. Those distributed systems that carry full history often can't deal with this fast enough or in any space efficient way. arch was an example of this. It had a retarded mechanism that forced users to care about caching certain revisions to speed it up , instead of doing it on it's own. I've never tried converting this repo to bazaar-ng, it wasn't far enough along when i started. It also had no cvs2bzr type program, and we aren't about to lose all our history. Except for monotone (builtin cvs_import) and subversion (cvs2svn), none of the cvs2* programs i've run across either run in reasonable time (those that don't actually understand how to extract rcs revisions would take weeks to convert our repo, literally), or could handle all the complexities a repository with our history present (branches of branches, etc). Most simply crash in weird ways or run out of memory :). Anyway: Monotone took 45 minutes just to diff two old revisions that are one revision away from each other. CVS takes about 2 minutes for the same operation. SVN on fsfs takes 4 seconds. The converted SVN repo has > 100000 revisions, and is only ~15% bigger than the cvs repo (mostly due to stupid copies it has to do to handle some tag fuckery people were doing in some cases. If it had been subversion from the start, it would have been smaller). We have cvs2svn speedup patches that were done with the KDE folks that make cvs2svn io bound again instead of cpu bound (it was O(N^2) in extracting cvs revision texts before). It takes 17 hours to convert the gcc repository now, only 45 minutes of cpu time :). It used to take 52 hours. I've also talked with Linus about version control before. He believes extreme distributed development is the way to go. I believe heavily that in most cases where you have a mix of corporations and free developers, it ends up causing people to "hide the ball" more than they should. This is particularly prevalent in GCC. We don't want design and development done and then sent as mega-patches presented as fait accompli, then watch these people whine as their designs get torn apart. We'd rather have the discussion on the mailing list and the work done in a visible place (IE CVS branch stored in some central place) rather than getting patch bombs. As a result (and there are many other reasons, i'm just presenting one of them), we actually don't *want* to move from a centralized model, in order to help control the social and political problems we'd have to face if we went fully distributed. Python may not face any of these problem to the degree that GCC does (i doubt many projects do actually. GCC is a very weird and tense political situation :P), because of size, etc, in which case, a distributed model may make more sense. However, you need to be careful to make sure that people understand that it hasn't actually changed your real development process (PEP's, etc), only the workflow used to implement it.

3 4

Fwd: Distributed RCS
by Guido van Rossum 15 Aug '05

15 Aug '05

Another fwd, describing how Steve Alexander's group user bazaar. --Guido van Rossum (home page: http://www.python.org/~guido/) ---------- Forwarded message ---------- From: Steve Alexander <steve(a)canonical.com> Date: Aug 12, 2005 4:00 PM Subject: Re: Distributed RCS To: Guido van Rossum <gvanrossum(a)gmail.com> Cc: Mark Shuttleworth <mark(a)canonical.com>, Martin Pool <mbp(a)canonical.com>, Fredrik Lundh <fredrik(a)pythonware.com> Hi Guido, I'm not going to post to python-dev just now, because I'm leaving on 1.5 weeks vacation tomorrow, and I'd rather be absent than unable to answer questions promptly. Martin Pool will be around next week, and will be able to take part in discussions on the list. Feel free to post all or part of Mark's or my emails to the python lists. Mark wrote: > > I hope that puts bazaar into perspective for you. Give it a spin - the > development 2.x codebase is robust enough now to handle a line of > development and do basic merging, we are switching our own development > to the pre-release 2.x line in October, and we will switch over all the > public archives we maintain in around March next year. A large part of the internal development at Canonical is the Launchpad system. This is about 30-40 kloc of Python code, including various Twisted services, cron scripts, a Zope 3 web application, database tools, ... It's being worked on by 20 software developers. Everyone uses bazaar 1.4 or 1.5, and around October, we'll be switching to use bazaar 2.x. I'll describe how we work on Launchpad using Bazaar. This is all from the Bazaar 1.x perspective, and some things will become simpler when we change to using Bazaar 2.x. I've left the description quite long, as I hope it will give you some of the flavour of working with a distributed RCS. == Two modes of working: shared branches and PQM == Bazaar supports two different modes of working for a group like the Launchpad team. 1. There's a shared read/write place that all the developers have access to. This is contains the branches we release from, and represents the "trunk" of the codebase. 2. A "virtual person" called the "patch queue manager" (PQM) has exclusive write access to a collection of branches. PQM takes instructions as GPG signed emails from launchpad developers, to merge their code into PQM's branches. We use the latter mode because we have PQM configured not only to accept requests to merge code into PQM's codebase, but to run all the tests first and refuse to merge if any test fails. == The typical flow of work on Launchpad == Say I want to work on some new feature for Launchpad. What do I do? 1. I use 'baz switch' to change my working tree from whatever I was working on last, and make it become PQM's latest code. baz switch rocketfuel(a)canonical.com/launchpad--devel--0 "rocketfuel" is the code-name for the branches we release our code from. PQM manages the rocketfuel branches. In Bazaar 1.x, collections of branches are called "archives" and are identified by an email address plus some other optional information. So, "rocketfuel(a)canonical.com" is PQM's email address. "launchpad--devel--0" is simply the name of the main launchpad branch. The format of branch names is very strict in Bazaar 1.x. It is much more open in Bazaar 2.x. 2. I use 'baz branch' to create my own branch of this code that I can commit changes to. baz branch steve.alexander(a)canonical.com/launchpad--ImproveLogins--0 My archive is called "steve.alexander(a)canonical.com". The branch will be used to work on the login functionality of Launchpad, so I have named the branch "launchpad--ImproveLogins--0". 3. I hack on the code, and from time to time commit my changes. I need to 'baz add' new files and directories, and 'baz rm' to remove files, and 'baz mv' to move files around. # hack hack hack baz commit -s "Refactored the whatever.py module." # hack hack hack baz del whatever_deprecated.py baz commit -s "Removed deprecated whatevers." # hack hack hack 4. Let's say I hacked on some stuff, but I didn't commit it. I don't like what I did, and I want to start again. # hack hack hack baz undo 'baz undo' puts the source code back into the state it was in after the last commit, and puts the changes somewhere. If I change my mind again, I can say 'baz redo', and get my changes back. 5. All this hacking and committing has been happening on my own workstation, without a connection to the internet. Perhaps I've been on a plane or at a cafe. When I have a connection again, I can make my work available for others to see by mirroring my code to a central location. Each Launchpad developer has a mirror of the archive they use for Launchpad work on a central machine at the Canonical data centre. In our case, the mirror command uses sftp to copy the latest changes I have made into the mirror on this central server. baz archive-mirror 6. Because we have a strict code review proccess for Launchpad development, I can't (or rather, shouldn't) submit my changes to PQM yet. I should get it reviewed. But, let's say Andrew wants to do some work that depends on my work, before my work has made its way into PQM's rocketfuel "Trunk". He can simply merge from me. # in Andrew's working tree, on his workstation. baz merge steve.alexander(a)canonical.com/launchpad--ImproveLogins--0 baz commit -s "Merged steve's ImproveLogins work." When Andrew eventually gets his work reviewed, and sends it on to PQM to be merged into Rocketfuel, the Bazaar merging algorithms will work out that Andrew merged from me, and will sort things out. Of course, there can be conflicts when people have worked in divergent ways on the same code. These are resolved in a similar way to CVS or SVN. 7. I want to get my code reviewed by a member of the review team. I add the details of my branch to the PendingReviews page on the launchpad development wiki. This wiki is publicly readable. https://wiki.launchpad.canonical.com/PendingReviews There is a script that periodically reads the PendingReviews page, attempts to merge the branches listed there into rocketfuel (just as PQM would do), and produces a diff for use by the review team. The diff represents what changes would be made to the rocketfuel Trunk were the branch in question to be sent to PQM. This diff is often enough for the reviewers to work with. If they need to see more context, they can simply check out the branch in question using 'baz get branchname'. The script also highlights whether there were any conflicts that would prevent a merge, and gives an indication of the size of the change. The script's output is accessible only to Launchpad developers. However, I've made a couple of screenshots to give you some idea of what it looks like. This is the summary page, that uses information taken from the PendingReviews wiki page. http://people.ubuntu.com/~stevea/branch-summary.png This is a typical diff representing what is to be merged. http://people.ubuntu.com/~stevea/branch-diff.png The reviewer sends an email to the author of the code, cc the launchpad-reviews mailing list. The review email typically has sections of code included, each line prefixed with '> ', with comments, questions and requests for improvement beneath each section of code. The reviewer will either approve the code for merging, approve the code providing certain remedial actions are taken, or reject the code, requiring a new review later. 8. My code has been successfully reviewed by JamesH, so I send a signed mail to PQM asking to merge my work into rocketfuel. submit-merge "r=JamesH, Improvements to logging in." pqm(a)pqm.ubuntu.com PQM checks that each merge request has r=someone in the message, as a reminder that launchpad developers need to have their code reviewed. The submit-merge script gets takes the archive name, the branch name, and the "patch level" that the branch is at, composes an email saying "pqm, please merge steve.alexander(a)canonical.com/launchpad--ImproveLogins-0--patch-18 into rocketfuel(a)canonical.com/launchpad--devel--0." Signs it with my gpg key, and mails it. Some time later, once PQM has merged the code and successfully run all the launchpad tests, an email will go out to me, and to a pqm-commits mailing list, saying that the merge was successful. If it was unsuccessful, I get an email with the error output. An irc robot listens to the pqm-commits mailing list, and announces new landings to the rocketfuel Trunk on irc. == Naming branches == The Launchpad team is distributed around the world. To cope with this, and also to get our community of users involved in the development of the software, Launchpad development emphasises writing specifications and proposals, and implementing features based on these proposals. You can read all the launchpad proposals on the launchpad development wiki. https://wiki.launchpad.canonical.com/ So, we usually name branches after the specification that is being implemented on that branch. The branch is named near the top of the specification, so someone reading the specification who has access to the source code can see what's happening with the implementation. Branches are also often named after bugs. For example, launchpad--bug123--0. The use of '--' in branch names, and the '--0' thing at the end is occassionally useful, but more of a hangover from the 'tla' system that bazaar is based on. This strict branch naming format is not being carried over into bazaar 2. == External contributors == The source code to Launchpad is not available at this time. We intend to make it open source at some point in the future, but I'm not sure when that will be. Let's consider what would happen if we decided to make the Launchpad code fully open source tommorrow. Someone from outside of Canonical could get a copy of the main launchpad "rocketfuel" branch, make their own branch by branching from the rocketfuel branch, do a bunch of work, mirror it to their own website, and email a Canonical launchpad developer to ask that it be reviewed, or merged into that launchpad developer's branch. This way, even though an outside contributor doesn't have rights with PQM, they could still make fine-grained commits, merge frmo a variety of places, and participate at the same level as someone employed by Canonical. -- Steve Alexander

3 2

xml.parsers.expat no userdata in callback functions
by Kristian Benoit 14 Aug '05

14 Aug '05

In the C version of expat, handlers receive a void *userdata, but it is not the case in the python version. This means one cant parse multiple files at the same time using the same handlers. You cant pass the context current context to the handler, you must base your code on global variables which is not so nice. Thanks Please leave the cc in the mail header as I'm not subscribed to the list. Kristian

3 2

new PEP type: Process
by David Goodger 13 Aug '05

13 Aug '05

Barry Warsaw and I, the PEP editors, have been discussing the need for a new PEP type lately. Martin von Löwis' PEP 347 was a prime example of a PEP that didn't fit into the existing Standards Track and Informational categories. We agreed upon a new "Process" PEP type. For more information, please see PEP 1 (http://www.python.org/peps/pep-0001.html) -- the type of which has also been changed to Process. Other good examples of Process PEPs are the release schedule PEPs, and I understand there may be a new one soon. (Please cc: any PEP-related mail to peps(a)python.org) -- David Goodger <http://python.net/~goodger>

2 2

Hosting svn.python.org
by skip＠pobox.com 13 Aug '05

13 Aug '05

martin> Log Message: martin> Add wush.net hosting. ... martin> + * Greg Stein suggested http://www.wush.net/subversion.php. ... I will enthusiastically cast my vote for tummy.com, Sean Reifschneider's company. Mojam leases a server there (Mojam & Musi-Cal websites running CentOS 4, Apache+mod_perl, Python, Mason, MySQLdb, Mailman, etc). Their service has been absolutely awesome. Sean is one of the python.org webmasters to boot, so he knows our culture very well already. Skip

2 1

Discussion draft: Proposed Py2.5 C API for set and frozenset objects
by Raymond Hettinger 13 Aug '05

13 Aug '05

The object and types -------------------- PySetObject subclass of object used for both sets and frozensets PySet_Type a basetype PyFrozenSet_Type a basetype The type check macros --------------------- PyFrozenSet_CheckExact(ob) a frozenset PyAnySet_CheckExact(ob) a set or frozenset PyAnySet_Check(ob) a set, frozenset, or subclass of either The constructors ---------------- obj PySet_New(it) takes an iterable or NULL; returns new ref obj PyFrozenSet_New(it) takes an iterable or NULL; returns new ref The fine grained methods ------------------------ int PySet_Size(so) int PySet_Contains(so, key) 1 for yes; 0 for no; -1 for error raises TypeError for unhashable key does not automatically convert to frozenset int PySet_Add(so, key) 0 for success; -1 for error raises TypeError for unhashable key raises MemoryError if no room to grow obj PySet_Pop(so) return new ref or NULL on failure raises KeyError if set is emtpy int PySet_Discard(so, key) 1 if found and removed 0 if not found (does not raise KeyError) -1 on error raises TypeError for unhashable key does not automatically convert to frozenset Course grained methods left for access through PyObject_CallMethod() -------------------------------------------------------------------- copy, clear, union, intersection, difference, symmetric_difference, update, intersection_update, difference_update, symmetric_difference_update issubset, issuperset, __reduce__ Other functions left for access through the existing abstract API ----------------------------------------------------------------- PyObject_RichCompareBool() PyObject_Hash() PyObject_Repr() PyObject_IsTrue() PyObject_Print() PyObject_GetIter()

1 0

plans for 2.4.2 and 2.5a1
by Anthony Baxter 13 Aug '05

13 Aug '05

So I'm currently planning for a 2.4.2 sometime around mid September. I figure we cut a release candidate either on the 7th or 14th, and a final a week later. In addition, I'd like to suggest we think about a first alpha of 2.5 sometime during March 2006, with a final release sometime around May-June. This would mean (assuming people are happy with this) we need to make a list of what's still outstanding for 2.5. There's a bunch of accepted PEPs that are waiting for code. Once that's done, there will be a final 2.4.3 sometime after or close to the 2.5 final release. Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

7 7

Re: [Python-Dev] plans for 2.4.2 and 2.5a1
by Raymond Hettinger 13 Aug '05

13 Aug '05

[Josiah] > At this point, I'd be happy to get > /any/ mechanism, with a preference to struct and/or binascii Assign 1023290 to me and I'll get it done in the next month or so. Raymond

1 0

Re: [Python-Dev] Exception Reorg PEP revised yet again
by Raymond Hettinger 13 Aug '05

13 Aug '05

> > Then I don't follow what you mean by "moved under os". > > In other words, to get the exception, do ``from os import > WindowsError``. Unfortunately we don't have a generic win module to > put it under. Maybe in the platform module instead? -1 on either. The WindowsError exception needs to in the main exception tree. It occurs in too many different modules and applications. That is a good reason for being in the main tree. If the name bugs you, I would support renaming it to PlatformError or somesuch. That would make it free for use with Mac errors and Linux errors. Also, it wouldn't tie a language feature to the name of an MS product. Raymond

6 8

set.remove feature/bug
by Paolino 13 Aug '05

13 Aug '05

I can't contact sourceforge bug tracker sorry. set.remove is trying to freeze sets when they are used as keys.No matter if an __hash__ method is defined. This is incoherent with Set.remove and dict.__delete__ & co. If this is a feature ,then I ask strongly to keep sets module in the stdlib for ever. Or if there is a workaround, please tell me here because python-list didn't help. class H(set): def __hash__(self):return id(self) s=H() f=set() f.add(s) f.remove(s) # this fails Regards Paolino

2 1