Status of the Derby, and request for another slip
The Great Argument Clinic Conversion Derby has been underway for about 2.5 weeks. Here's a status update. I'll try to keep this short.
It's taking a lot longer than I expected it to. This has been due to
- bugs and flaws in Argument Clinic itself (a "flaw" is an "I didn't foresee that" design error),
- having to make changes to CPython itself, which required *very* careful work and was therefore quite time consuming, and
- my time being a huge bottleneck.
I'd like to extend the Derby by two more weeks and add a fourth beta.
--
I've made huge improvements to the implementation of Argument Clinic, in
response to bugs filed and crippling-missing-features requested.
Argument Clinic is growing more sophisticated by the day.
I've also, regrettably, made some modifications to CPython, to better support signatures in builtins. In particular, I had to modify four additional callable types to add support for "__text_signature__", including PyTypeObject. These changes were reviewed by Nick and Guido, so I have a high confidence that the changes will not destabilize Python 3.4. (If there is demand, I can compile a list of all such arguably-not-a-bugfix post-feature-freeze changes to Python 3.4 if there's interest.)
The main problem with running the Derby: everything is gated on me. Until just recently, I've had to review every patch, and I've had to make every modification to clinic.py. I've been spending all day, every day, with very little time off, on the Derby, and still it's nowhere near enough.
This is changing, slowly. I now have a handful of people who make small changes to clinic.py or review patches. But major brain surgery on the tool is still up to me, and I still have a half-week's worth of work on my to-do list.
I've been prioritizing bug fixes and crippling missing features over reviewing patches. (I figured that would maximize the productivity of the Derby participants.) As a result, I have a patch backlog that would makes you cry. I've been reviewing patches FIFO, and the patch waiting at the front of my queue is ten days old.
This means that a lot of the conversion work done for the Derby has not been checked in yet.
To be honest I have absolutely no idea what % of files have been converted so far. Figuring it out would use time that would be better spent catching up on my to-do list.
--
I'd like to continue the Derby for two more weeks and insert a fourth beta. That would put the release date for Python 3.4 final on March 31st.
With the new conversion policy in place (see my recent posting to python-dev), the number of new surprises that I would have to spend time on should drop to about zero. (My new answer: "That will have to wait until 3.5.") I have about a half-week's worth of bug fixes / missing features work for clinic.py right now; once that's done I'll be able to spend nearly all of my time on code reviews. (If I catch up, I might even start doing conversions myself!) So I expect the pace of the Derby to ramp up in the coming week.
I *would* ask for three more weeks, not two, just to be dead certain we finish before rc1. But then the release date for 3.4 final would be April 7th, two days before the start of PyCon US 2014, and I don't want to shave it that close. If we had to slip again for some reason, we'd be cutting the release *during* PyCon US, and I want to avoid *that* at all costs, particularly as that may be impractical for the platform release manager (their build environment may not be portable). I want 3.4 out before PyCon US so that trunk is open for 3.5 during the dev sprints.
The alternative would be: stop all submissions to the Derby right now, iterate (quickly!) on the existing patches, check them in, and proceed to rc1.
In theory, as release manager I could simply announce we were doing this. However, I only want to continue doing this if I have approval from the core dev community. If slipping the schedule again means a majority of people withdraw their support for the Derby, then it would probably be best to cut our losses now.
Please vote for either "continue the Derby" (which also means slipping the schedule and adding a fourth beta) or "stop the Derby".
Thank you for your attention,
//arry/
On sam., 2014-01-25 at 05:09 -0800, Larry Hastings wrote:
Please vote for either "continue the Derby" (which also means slipping the schedule and adding a fourth beta) or "stop the Derby".
Stop the Derby. We needn't convert everything for 3.4 (I didn't even know that was your goal).
Regards
Antoine.
On 01/25/2014 05:35 AM, Antoine Pitrou wrote:
Stop the Derby. We needn't convert everything for 3.4 (I didn't even know that was your goal).
Converting everything was my goal at one point. At this point it is nowhere near viable, not the least because there simply isn't enough time. As discussed yesterday on python-dev (subject: "Argument Clinic: what to do with builtins with non-standard signatures?"), and today (subject: "New policies for the Derby -- please read!"), there are large numbers of functions in Python 3.4 that have signatures that can't be expressed by inspect.Signature without semantic changes. Since we can't make those changes for 3.4, we can't convert them to Argument Clinic yet.
//arry/
On 25 January 2014 23:49, Larry Hastings <larry@hastings.org> wrote:
On 01/25/2014 05:35 AM, Antoine Pitrou wrote:
Stop the Derby. We needn't convert everything for 3.4 (I didn't even know that was your goal).
Converting everything was my goal at one point. At this point it is nowhere near viable, not the least because there simply isn't enough time. As discussed yesterday on python-dev (subject: "Argument Clinic: what to do with builtins with non-standard signatures?"), and today (subject: "New policies for the Derby -- please read!"), there are large numbers of functions in Python 3.4 that have signatures that can't be expressed by inspect.Signature without semantic changes. Since we can't make those changes for 3.4, we can't convert them to Argument Clinic yet.
As per my post to python-dev, the main thing I would like to see postponed to 3.5 is the varargs support in Argument Clinic. It sounds like that is worth postponing until 3.5 due to the impact it has on AC itself, rather than the other issues which need to be delayed due to their impact on the inspect module and other public APIs.
Basically, I suggest implementing the current bug fixes & minor features AC patch you're working on, and then calling it done for 3.4, with only outright bugs in existing conversions to be fixed before 3.5. If it's "I can't convert function X because AC doesn't have feature Y or suffers from bug Z", then that means function X doesn't get converted until 3.5. Be ruthless on the "Argument Clinic doesn't support that yet, so it will have to wait to be converted" :)
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. That patch is done, it just needs a couple of tests cleaned up and a review before it can be merged.
Once __text_signature__ is made a public API in Python 3.5, then it also opens up lots of opportunities for easier introspection support in other implementations, not just in CPython.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On dim., 2014-01-26 at 01:40 +1000, Nick Coghlan wrote:
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. That patch is done, it just needs a couple of tests cleaned up and a review before it can be merged.
I don't think this is a killer feature that would deserve an extension of the release cycle. The feature freeze period is meant for bug fixing, not to shoehorn the latest bangs and whistles. Normally the whole Derby thing should already have waited for 3.5; in hindsight it would probably have been better to do just that.
Regards
Antoine.
On sam., 2014-01-25 at 17:14 +0100, Antoine Pitrou wrote:
On dim., 2014-01-26 at 01:40 +1000, Nick Coghlan wrote:
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. That patch is done, it just needs a couple of tests cleaned up and a review before it can be merged.
I don't think this is a killer feature that would deserve an extension of the release cycle. The feature freeze period is meant for bug fixing, not to shoehorn the latest bangs and whistles.
(perhaps it should be "bells and whistles"; I apologize for the mismatch)
Regards
Antoine.
On 26 January 2014 02:14, Antoine Pitrou <solipsis@pitrou.net> wrote:
On dim., 2014-01-26 at 01:40 +1000, Nick Coghlan wrote:
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. That patch is done, it just needs a couple of tests cleaned up and a review before it can be merged.
I don't think this is a killer feature that would deserve an extension of the release cycle. The feature freeze period is meant for bug fixing, not to shoehorn the latest bangs and whistles. Normally the whole Derby thing should already have waited for 3.5; in hindsight it would probably have been better to do just that.
This is a fair point - I think it would also be reasonable (and quite possibly preferable) to say "no more clinic conversions for 3.4", including my almost complete patch for the builtins and any other almost complete patches. The existing conversions would stay, but our focus for the remainder of the release cycle could then return to bug fixes for existing changes (including any genuine bugs in AC for the already converted extension modules.
I was OK with trying the conversion within the beta period, but I think the experience of the past couple of weeks has shown that it was higher impact than we first thought, and that there were more gaps in the current capabilities of both argument clinic and the inspect module than we first thought.
By postponing further conversions, we can refocus on the 3.4 release in the near term, and then update PEP 457 for 3.5 based on our experience in attempting these conversions for 3.4.
It wouldn't be the first time where a change we've made in one release has mostly been about setting up a benefit for end users that wasn't fully realised until the next release (__qualname__ in 3.3 leading to unbound method pickling support in 3.4 is the main recent example that comes to mind, but PEP 451 certainly has follow on work that will continue into 3.5, and the underlying goal of adding functools.singledispatch was actually to convert the pprint module to use it, which now won't be happening until 3.5 at the earliest).
While part of my brain is still saying "but we're so close...", I think I'm coming around to the view that Antoine is right and we should just stop the Derby and focus on getting the 3.4 release candidate ready for February 9. It's disappointing, definitely, but it's also the most responsible course of action available.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
25.01.14 18:14, Antoine Pitrou написав(ла):
I don't think this is a killer feature that would deserve an extension of the release cycle. The feature freeze period is meant for bug fixing, not to shoehorn the latest bangs and whistles. Normally the whole Derby thing should already have waited for 3.5; in hindsight it would probably have been better to do just that.
There is at least one benefit from Derby: it have exhibited many errors in already clinicalized code and in pydoc and inspect modules (and in Argument Clinic itself).
On 26 January 2014 22:57, Serhiy Storchaka <storchaka@gmail.com> wrote:
25.01.14 18:14, Antoine Pitrou написав(ла):
I don't think this is a killer feature that would deserve an extension of the release cycle. The feature freeze period is meant for bug fixing, not to shoehorn the latest bangs and whistles. Normally the whole Derby thing should already have waited for 3.5; in hindsight it would probably have been better to do just that.
There is at least one benefit from Derby: it have exhibited many errors in already clinicalized code and in pydoc and inspect modules (and in Argument Clinic itself).
Right, I think the Derby provided valuable experience, but I also now think Antoine's right that we should postponing any further conversions until 3.5, and focus on cleaning up both the existing conversions, as well as the technical details of working with Argument Clinic.
Specifically, I think we want to have the following in place for 3.4:
- Larry's patch to make the Argument Clinic signature markers more robust (http://bugs.python.org/issue20326)
- Generating separate clinic files by default
- Integrating clinic into "make patchcheck", similar to the reindent.py integration (http://bugs.python.org/issue20264)
- Adding a commit hook that ensures clinic files have been regenerated prior to commit/push
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 01/25/2014 07:40 AM, Nick Coghlan wrote:
Basically, I suggest implementing the current bug fixes & minor features AC patch you're working on, and then calling it done for 3.4, with only outright bugs in existing conversions to be fixed before 3.5. [...]
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. [...]
+1 to what he said.
-- ~Ethan~
On 25/01/14 19:33, Ethan Furman wrote:
On 01/25/2014 07:40 AM, Nick Coghlan wrote:
Basically, I suggest implementing the current bug fixes & minor features AC patch you're working on, and then calling it done for 3.4, with only outright bugs in existing conversions to be fixed before 3.5. [...]
I think an extra beta is still a good idea, though - I'd like to get at least the builtins that are already supported converted, since that clearly demonstrates the benefits the tool is designed to bring in reducing the discrepancies between extension code and pure Python code. [...]
+1 to what he said.
+1. I agree too.
-- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Larry Hastings <larry@hastings.org> wrote:
Please vote for either "continue the Derby" (which also means slipping the schedule and adding a fourth beta) or "stop the Derby".
I think releasing 3.4 on time is more important than adding internal changes that most people won't see at all. So I vote for releasing according to the schedule.
Stefan Krah
On Sat, Jan 25, 2014 at 3:09 PM, Larry Hastings <larry@hastings.org> wrote:
To be honest I have absolutely no idea what % of files have been converted so far. Figuring it out would use time that would be better spent catching up on my to-do list.
Of the 22 Derby issues:
- 2 have been closed: 20133 (audioop) and 20151 (binascci)
- 7 are practically done
- 6 are about half done
- 7 still need most (or all) of the work done
So I'd say that the derby is about half finished, not taking into account the required changes and improvements to clinic.py and additional overhead and unexpected problems.
Details on the 20 open Derby issues:
Apparently or nearly completed Derby issues (7):
- 20148: _sre
- 20168: tkinter
- 20172: Windows stuff
- 20173: codecs, md5, sha*
- 20174: socket (+ functools)
- 20178: ctypes, sqlite (+ _curses_cpanel.c)
- 20180: collections, itertools, random, unicode, stringlib (+ xxlimited.c, xxmodule.c, xxsutype.c)
Approximately half completed Derby issues (6):
- 20152: cmath, multibytecodec, array, pyexpat, fcntl, pwd, spwd, grp
- 20159: etree, PC/_msc.c, PC/bdist_wininst/install.c
- 20171: curses (117 sites!)
- 20177: time, datetime
- 20182: select, signal, zlib, sys (+ _hashopenssl.c)
- 20185: type, long, list, float, resource, thread, bz2, bisect, marshal, exceptions, nis, gc, warnings, crypt
Derby issues not started or barely started (7):
- 20170: posix (137 sites!)
- 20175: io (+ multiprocessing)
- 20179: ssl, overlapped (+ bytesobject.c, bytes_methods.c)
- 20181: bytearray, parser, unicodedata, readline
- 20183: mmap, locale, zipimport, ossaudiodev, _testbuffer.c, struct
- 20184: bltinmodule, faulthandler, pickle, lzma, termios, syslog, dbm, gdbm, json
- 20186: tuple, memory, descr, complex, operator, opcode, lsprof, heapq, weakref, struct, range, object, modules, func, file, enum, code, bool, symtable, math, _tracemalloc.c, io, csv
Please vote for either "continue the Derby" (which also means slipping the schedule and adding a fourth beta) or "stop the Derby".
Having worked intensively on the Derby for over a week, I feel an urge to see it through soon. The least we could do is to stop adding features to clinic.py, only fixing bugs; convert everything that is possible to convert now and mark everything else appropriately; and leave tying up the loose ends for 3.5.
However, I'm not a "core dev" and can certainly see the problem with pushing back the release date again, with significant uncertainty whether even that will be enough time. Therefore, I'm not voting.
I can, however, promise to convert a great deal more code in the next two weeks if it is decided to push the release.
- Tal
On 26 January 2014 23:15, Tal Einat <taleinat@gmail.com> wrote:
Having worked intensively on the Derby for over a week, I feel an urge to see it through soon. The least we could do is to stop adding features to clinic.py, only fixing bugs; convert everything that is possible to convert now and mark everything else appropriately; and leave tying up the loose ends for 3.5.
Actually, I think one thing we need to consider is when Larry plans to make the maintenance branch for 3.4. I believe historically, we have done that with the first release candidate, so the current schedule would allow clinic conversions for 3.5 to resume in just couple of weeks time. The difference between the beta and RC period is that only critical regressions discovered in the RC are considered for inclusion
- even normal bug fixing is typically postponed until after the release at that point. Any fixes incorporated during that period typically involve at least three core devs (release manager to say yes to including the fix, one to write a patch, another to review it).
Cheers, Nick.
P.S. There are a couple of builtin docstring fixes I now plan to apply directly, since I won't be converting those to Argument Clinic for 3.4 after all.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 25, 2014, at 5:09 AM, Larry Hastings <larry@hastings.org> wrote:
I'd like to extend the Derby by two more weeks and add a fourth beta.
There's no harm in adding a fourth beta and if it improves the release you should probably do so. We don't owe anyone a particular release date and AFAICT no one is holding their breath. The release date is a self-imposed deadline.
That said, further 3.4 work on the Argument Clinic ought to be aimed at improving the quality of the work so far, rather than digging in deeper.
When it comes to conceptual flaws, the important thing is to make sure that the 3.4 release doesn't lock you in to a decision you're going to want to change later.
FWIW, several lines of the Zen of Python seem to be relevant to this discussion.
Raymond
Am 25.01.2014 14:09, schrieb Larry Hastings:
I'd like to extend the Derby by two more weeks and add a fourth beta.
In the past I did like the way how accurate the releases were planned and didn't slip at all, or only slip for a few days.
This may sound a bit selfish, but a few Ubuntu developers do run their own Derby to use 3.4 as the default Python3 version for the next Ubuntu release in April, so a timely release would be appreciated ;)
Please vote for either "continue the Derby" (which also means slipping the schedule and adding a fourth beta) or "stop the Derby".
that would be a biased -1 from my side
Matthias
PS: For those interested, Ubuntu currently has the 3.4 beta as an alternative version built. I hope to have better results next week of what will break with 3.4 as the default python3 version.
On Sun, Jan 26, 2014 at 9:53 AM, Matthias Klose <doko@ubuntu.com> wrote:
Am 25.01.2014 14:09, schrieb Larry Hastings:
I'd like to extend the Derby by two more weeks and add a fourth beta.
In the past I did like the way how accurate the releases were planned and didn't slip at all, or only slip for a few days.
This may sound a bit selfish, but a few Ubuntu developers do run their own Derby to use 3.4 as the default Python3 version for the next Ubuntu release in April, so a timely release would be appreciated ;)
This actually sounds like a real and interesting reason to have stable Python 3.4 out in march. Ubuntu 14.04 is a LTS, which makes it an important version for corporate distributions.
In addition, other major projects align themselves with released versions of Python. For example Django - I would love to see trunk Django formally support Python 3.4 ASAP.
Eli
On Jan 26, 2014, at 01:59 PM, Eli Bendersky wrote:
This actually sounds like a real and interesting reason to have stable Python 3.4 out in march. Ubuntu 14.04 is a LTS, which makes it an important version for corporate distributions.
Ubuntu 14.04 feature freeze is Feb 20, with beta 1 coming on Feb 27, so I think we have to make a hard decision about whether Python 3.4 will be the default before then (actually, Matthias and I will discuss it this week).
If we go with 3.4 as default, then 14.04 final beta is March 27th. Python 3.4's final release is currently schedule for March 16, but that doesn't include the proposed additional beta. Pushing everything back a week means the schedule is quite tight, so we may have to plan on doing an SRU (stable release upgrade) to the final Python 3.4. That's not ideal though because it means a lot of users may not get it (if they don't upgrade), and most probably won't see it until 14.04.1 is released some time later.
The closer Python 3.4 can stick to the March 16th release, the better.
-Barry
On Sun, Jan 26, 2014 at 2:16 PM, Barry Warsaw <barry@python.org> wrote:
On Jan 26, 2014, at 01:59 PM, Eli Bendersky wrote:
This actually sounds like a real and interesting reason to have stable Python 3.4 out in march. Ubuntu 14.04 is a LTS, which makes it an important version for corporate distributions.
Ubuntu 14.04 feature freeze is Feb 20, with beta 1 coming on Feb 27, so I think we have to make a hard decision about whether Python 3.4 will be the default before then (actually, Matthias and I will discuss it this week).
If we go with 3.4 as default, then 14.04 final beta is March 27th. Python 3.4's final release is currently schedule for March 16, but that doesn't include the proposed additional beta. Pushing everything back a week means the schedule is quite tight, so we may have to plan on doing an SRU (stable release upgrade) to the final Python 3.4. That's not ideal though because it means a lot of users may not get it (if they don't upgrade), and most probably won't see it until 14.04.1 is released some time later.
The closer Python 3.4 can stick to the March 16th release, the better.
Yes. That's the whole point of time-based releases. Please stick to the plan. We've already slipped once. Perfection is not required.
-- --Guido van Rossum (python.org/~guido)
On 01/26/2014 02:18 PM, Guido van Rossum wrote:
On Sun, Jan 26, 2014 at 2:16 PM, Barry Warsaw <barry@python.org <mailto:barry@python.org>> wrote:
The closer Python 3.4 can stick to the March 16th release, the better.Yes. That's the whole point of time-based releases. Please stick to the plan. We've already slipped once. Perfection is not required.
The trend was clear even before Guido weighed in. But that certainly clinches it.
I've just posted to python-dev announcing that the Derby is closing.
I'm not cutting it off instantly, in case there are people with
partially-complete patches that haven't been posted to the issue
tracker. (I know of one such person: me.) So I'm giving them a little
time to submit their patches, Wednesday morning at midnight.
This'll be nice for me as I could use a break from Argument Clinic, and the Derby in particular.
And, in hindsight, perfection was never on the table ;-)
//arry/
participants (13)
-
Antoine Pitrou -
Barry Warsaw -
Eli Bendersky -
Ethan Furman -
Guido van Rossum -
Jesus Cea -
Larry Hastings -
Matthias Klose -
Nick Coghlan -
Raymond Hettinger -
Serhiy Storchaka -
Stefan Krah -
Tal Einat