Mailman 3 Our failure at handling GSoC students - Python-Dev

Our failure at handling GSoC students

Antoine Pitrou

6 Aug 2013 6 Aug '13

7:26 p.m.

Hello, I would like to point out that we currently fail at handling GSoC projects and bringing them to completion. One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3 Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is). What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are. Regards Antoine.

Show replies by date

Eli Bendersky

6 Aug 6 Aug

7:43 p.m.

On Tue, Aug 6, 2013 at 12:26 PM, Antoine Pitrou wrote:

...

Hello,

I would like to point out that we currently fail at handling GSoC projects and bringing them to completion.

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber:

http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is).

What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are.

I would like to point out something that stands out in this list of issues: such a method of producing dozens of patches simultaneously is extremely unwise, unless there's a crucial piece of history I'm missing. It is much more prudent to start with one or two exemplary modules, and if those fully pass code review, send out patches for others. The reason is obvious - code review may turn up problems or requests for change. Going backwards to modify 57 patches is not something anyone would want to do. Eli

Antoine Pitrou

7:51 p.m.

On Tue, 6 Aug 2013 12:43:40 -0700 Eli Bendersky wrote:

...

On Tue, Aug 6, 2013 at 12:26 PM, Antoine Pitrou wrote:

...
Hello,

I would like to point out that we currently fail at handling GSoC projects and bringing them to completion.

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber:

http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is).

What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are.

I would like to point out something that stands out in this list of issues: such a method of producing dozens of patches simultaneously is extremely unwise, unless there's a crucial piece of history I'm missing. It is much more prudent to start with one or two exemplary modules, and if those fully pass code review, send out patches for others. The reason is obvious - code review may turn up problems or requests for change. Going backwards to modify 57 patches is not something anyone would want to do.

I definitely agree, but this is part of our failure too. A beginner contributor isn't supposed to know the best way to contribute if nobody tells him/her beforehand. Regards Antoine.

Fred Drake

8:07 p.m.

On Tue, Aug 6, 2013 at 3:51 PM, Antoine Pitrou wrote:

...

I definitely agree, but this is part of our failure too.

I'd say this is strictly our failure, not the students'. This isn't really a new problem, I don't think, though the shape of this collection of patches makes it obvious. I haven't been active with GSoC the last couple of years, but if we don't have any sort of guide for mentors, we probably should, and this is an issue that should be mentioned as one that requires discussion with the students. That's our role as a community and as mentors when it comes to GSoC. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

martin＠v.loewis.de

7 Aug 7 Aug

8:34 a.m.

Zitat von Eli Bendersky :

...

I would like to point out something that stands out in this list of issues: such a method of producing dozens of patches simultaneously is extremely unwise, unless there's a crucial piece of history I'm missing. It is much more prudent to start with one or two exemplary modules, and if those fully pass code review, send out patches for others. The reason is obvious - code review may turn up problems or requests for change. Going backwards to modify 57 patches is not something anyone would want to do.

Robin did exactly that: submit a few patches first, receive feedback, submit more patches. At the end of the project,he submitted his entire work. Regards, Martin

Alexander Belopolsky

8:54 a.m.

On Wed, Aug 7, 2013 at 4:34 AM, wrote:

...

Robin did exactly that: submit a few patches first, receive feedback, submit more patches. At the end of the project,he submitted his entire work.

That's not how the history looks on the tracker. Robin submitted ~50 patches before I suggested that "we should start with the "xx" modules." Then he did submit patches to the example modules, but have never responded to my reviews. http://bugs.python.org/issue15787 http://bugs.python.org/issue15848 http://bugs.python.org/issue15849

Ethan Furman

5:12 p.m.

On 08/07/2013 01:54 AM, Alexander Belopolsky wrote:

...

That's not how the history looks on the tracker. Robin submitted ~50 patches before I suggested that "we should start with the "xx" modules." Then he did submit patches to the example modules, but have never responded to my reviews.

Dumb question, but does he know how to publish his responses? It took me a week to figure that out. Of course, it would be up to him to ask why his responses weren't being acknowledged. (I'm speaking of the reitvald tool.) -- ~Ethan~

Alexander Belopolsky

7:43 p.m.

On Wed, Aug 7, 2013 at 1:12 PM, Ethan Furman wrote:

...

Dumb question, but does he know how to publish his responses? ... (I'm

speaking of the reitvald tool.) The patches that I reviewed: #15390 (datetime), #15848 (xxsubtype), and #15849 (xxmodule) did not have Reitvald "review" links. I reviewed them in the tracker comments.

Skip Montanaro

6 Aug 6 Aug

7:45 p.m.

...

It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it...

Given that Python development is done using a good DVCS now, it seems that if each manageable chunk of changes is done on a separate branch, the likelihood of acceptance of any one change goes way up (as it's much easier to analyze in isolation), and the likelihood that one small change nukes the entire collective patch goes way down. I don't know if that will address all concerns and improve the success rate, but I would personally find it easier to process 100 changes, each with 37 patch chunks than one change having 3700 chunks. Smaller haystacks make it easier to find the needles. In addition, there should be less pressure for someone to analyze the entire lot. If you get burned out at change 12, others should be there to pick up from change 13 without having to start over, re-analyzing changes 1 through 12. Skip

Terry Reedy

10:34 p.m.

On 8/6/2013 3:26 PM, Antoine Pitrou wrote:

...

I would like to point out that we currently fail at handling GSoC projects and bringing them to completion.

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is).

What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are.

There are 2 GSOC students working on Idle tests (mentored by Todd Rovito). Each file tested is a separate issue and separate patch. I have fallen behind reviewing them because of unexpected issues first with Idle and then with buildbots, but have been able to make some comments and some commits. I plan to do more before they disappear, and to get to everything eventually. -- Terry Jan Reedy

Todd V Rovito

10:36 p.m.

On Aug 6, 2013, at 3:26 PM, Antoine Pitrou wrote:

...

I would like to point out that we currently fail at handling GSoC projects and bringing them to completion. In the past I have noticed the same thing with IDLE. Students and mentors act outside of the standard Python development process then the final student products never get committed.

...

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3 I agree this is a sad example.

...

What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker). It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know....

So for this year I designed an IDLE project that specifically forced the students to be like normal contributors and use the standard Python development model. See this link for the project description: http://wiki.python.org/moin/SummerOfCode/2013/python-core From the project description: "Successful student proposals should not under estimate how long it takes to get code committed to CPython. A student must be able to concisely communicate and document the unit test framework's design to the Python community in order to get the framework committed to the CPython source tree. Do not underestimate how much time this communication and documentation will actually take in your proposal!!! Often times it will take several passes and several code reviews for a patch to get committed into CPython. This project is approximately 40% coding and 60% communication. This project requires average Python coding skills with excellent communication skills and a unrelenting persistence to get this job done to the satisfaction of at least one Python Core Developer so the work will be committed into the CPython source tree." To date the students have gotten three commits completed and seven total issues opened. Here is a google spreadsheet with the details: https://docs.google.com/spreadsheet/lv?key=0AqHo248BJw3RdFRnREo5TGtrQmxvQi1o... It is too early to tell how effective the students have been. I do wish more unit tests were created but it all takes time to convince a core Python developer to make the commit (and with good reason). In this case Terry Reedy has been a huge help! I think the students are having fun and hopefully will stay involved for years to come.

Nick Coghlan

7 Aug 7 Aug

1:39 a.m.

On 7 August 2013 05:26, Antoine Pitrou wrote:

...

Hello,

I would like to point out that we currently fail at handling GSoC projects and bringing them to completion.

Agreed.

...

What didn't produce an alarm during Robin's work is that GSoC work is done in private. Therefore, other core developers than the mentor don't get to give an advice early, as would happen with any normal proposal done publicly (on the mailing-list or on the bug tracker).

This isn't the way GSoC is supposed to work. Mentors are supposed to nudge students towards the regular channels for the project. This may mean a sig (e.g. the import engine work a few years ago was discussed on import-sig. That didn't end up being committed, since Greg's work revealed some fundamental problems with the proposed architecture, but the knowledge wasn't restricted to just myself and Greg), or else a more general channel like core-mentorship or python-ideas. Ideally (and this isn't going to be possible for every GSoC project), mentors will be able to help break the project down into reviewable chunks proposed as incremental issues, rather than producing one big patch at the end of the summer.

...

It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are.

Indeed. I added some preliminary guidelines for mentors to the GSoC "Expectations" page: http://wiki.python.org/moin/SummerOfCode/Expectations#guidelines-for-mentors I also added a link to the expectations page from http://wiki.python.org/moin/SummerOfCode/2013#prospective-mentors Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Stephen J. Turnbull

5:06 a.m.

Nick Coghlan writes:

...

On 7 August 2013 05:26, Antoine Pitrou wrote:

...

...
I would like to point out that we currently fail at handling GSoC projects and bringing them to completion.

Agreed.

I have no opinion on that statement, having not looked at the projects.

...

...
What didn't produce an alarm during Robin's work is that GSoC work is done in private.

This isn't the way GSoC is supposed to work.

Indeed. I've seen this in one of the orgs I mentor for, and we may ask that mentor to go elsewhere if we don't get a credible promise to shape up. (That's an indication of how seriously my orgs take "work in public", not a suggestion for Python action in any case.)

...

or else a more general channel like core-mentorship or python-ideas.

+1 for python-ideas if there is no better fit in a specialist list, moving to python-dev as usual if the mentor judges the student sufficiently mature.[1] If the student's posting becomes annoying on python-ideas, the mentor should provide netiquette guidance. IMO the project-specific mentoring will become an annoyance on core-mentorship since it continues for the whole summer, and changing "preliminary" venues midstream doesn't seem like a great idea to me. My orgs require (in only one case successfully :-) weekly progress reports to the developers' list (per above, that would be python-ideas, not python-dev). In that successful case, GSoC is essentially the only content posted to that dev list, though. I'm not sure if that matters.

...

Ideally (and this isn't going to be possible for every GSoC project), mentors will be able to help break the project down into reviewable chunks proposed as incremental issues, rather than producing one big patch at the end of the summer.

GSoC suggests (at about the level of an RFC SHOULD) that students be committing early and often to a publicly accessible branch. I don't see a good reason why that wouldn't work even for complex projects that can't be merged until end of summer. (I wonder whether such projects should be used as GSoC tasks, as well, but as I haven't actually looked at Python GSoC tasks, I'll leave that in parens.) Footnotes: [1] By that I mean that I often observe students mixing blue-sky design with hard-core implementation details and necessary design revision late in the summer. If the project and student are "mature," that won't happen.

Lennart Regebro

5:20 a.m.

On Tue, Aug 6, 2013 at 9:26 PM, Antoine Pitrou wrote:

...

What didn't produce an alarm during Robin's work is that GSoC work is done in private.

Why is it done in private? //Lennart

martin＠v.loewis.de

8:44 a.m.

Zitat von Lennart Regebro :

...

On Tue, Aug 6, 2013 at 9:26 PM, Antoine Pitrou wrote:

...
What didn't produce an alarm during Robin's work is that GSoC work is done in private.

Why is it done in private?

It wasn't really done in private, not more than any other contribution. A PEP was accepted before the project even started. Regards, Martin

martin＠v.loewis.de

8:09 a.m.

Zitat von Antoine Pitrou :

...

One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber:

I personally dont consider it failed, yet. I still plan to integrate them, hopefully for 3.4.

...

Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is).

Choice of supporting PEP 384 was deliberate. It will change all types into heap types, which is useful for multiple-interpreter support and GC.

...

What didn't produce an alarm during Robin's work is that GSoC work is done in private.

It wasn't really done in private. Robin posted to python-dev, anybody who would have been interested could have joined discussions.

...

It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are.

It's indeed unfortunate that RL interfered with my Python contributions. I apologize for that. However, anybody who wanted to catch up could have contacted Robin or myself. As overworked as we all are, nobody did. Regards, Martin

Alexander Belopolsky

8:41 a.m.

On Wed, Aug 7, 2013 at 4:09 AM, wrote:

...

..

...
What didn't produce an alarm during Robin's work is that GSoC work is done in private.

It wasn't really done in private. Robin posted to python-dev, anybody who would have been interested could have joined discussions.

True. In addition, Robin's work was posted at bugs.python.org and received reviews.

...

However, anybody who wanted to catch up could have contacted Robin or myself. As overworked as we all are, nobody did.

Not true. See http://bugs.python.org/issue15787.

Antoine Pitrou

9:12 a.m.

Le Wed, 07 Aug 2013 10:09:16 +0200, martin@v.loewis.de a écrit :

...

...
Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is).

Choice of supporting PEP 384 was deliberate. It will change all types into heap types, which is useful for multiple-interpreter support and GC.

If I'm not mistaken, static C types shouln't benefit much from GC, since they only reference C functions. Also, PyType_FromSpec() makes reference counting delicate when subclasses are allowed.

...

...
What didn't produce an alarm during Robin's work is that GSoC work is done in private.

It wasn't really done in private. Robin posted to python-dev, anybody who would have been interested could have joined discussions.

I'm sorry if I misremembered how things happened. However, it's clear that the produced patches (including their number) cause problems for reviewers, and very few of them have been integrated. Regards Antoine.

3914

Age (days ago)

3915

Last active (days ago)

List overview

Download

17 comments

12 participants

participants (12)

Alexander Belopolsky
Antoine Pitrou
Eli Bendersky
Ethan Furman
Fred Drake
Lennart Regebro
martin＠v.loewis.de
Nick Coghlan
Skip Montanaro
Stephen J. Turnbull
Terry Reedy
Todd V Rovito

Our failure at handling GSoC students

tags

participants (12)