Anonymizing the PyCon review process

Last night I got a couple of PyCon talks rejected, and someone else sent me a rejection email they'd received. I wasn't surprised at the rejections, but I was quite surprised that many of the review comments were at least in part based on the presenter (sometimes the incorrectly assumed presenter) instead of on the proposed talk. Out of the 14 reviews, 5 of them have comments about the author of the proposal. I'll just give 2 examples here, taken from 2 different reviewers on 2 different proposals. 1. Imagine you're a relatively unknown Python programmer, you submit a talk to PyCon, and get back a review whose first sentence reads: I don't know the reputation of this particular speaker, so I won't "+1" That sends a pretty unfortunate message: Because you're not a recognized Python person, I wont give you the thumbs up. Maybe I'm being naive or simplistic, but I'd have hoped one route to becoming recognized would be by giving a PyCon talk. From the POV of the review recipient the leading justification for a non-recommendation has nothing to do with the talk! 2. What if you gave a PyCon talk in an earlier year that wasn't rated as highly as other talks? You send in a PyCon proposal, and get this back: I like XXX but honestly his talk at pycon 09 went poorly. That was the *entire* review in this case. What's the message here? Sounds like: well, you gave a talk once and it wasn't so great, so part of my vote against your proposal is because of that. It's like telling people to go away and not bother ever submitting again. Again, it's that's not based on the current talk proposal. People tend to get better at giving talks. If a proposal's content is technically good enough to get in, let them give another talk and help them to make it better. Just when is XXX supposed to re-send another PyCon talk, if ever? What makes this even worse is that XXX was not even the primary author of the proposal, and was not to be the speaker. So here we have a review that's negative *entirely* due to a talk that someone else gave in a previous year. How discouraging. Should the person in the future "take one for the team" and decline to be listed as a co-author on joint proposals - even though they're not going to speak - fearing that a reviewer will reply with a -1 and a one-line dismissal? That's the unfortunate dynamic that the above "review" has created. I hope this doesn't sound like personal sour grapes. It's not at all. I've had *tons* of rejection letters in my life (see http://bit.ly/1xytIr), including from PyCon. They're water off a duck's back at this point :-) I do however care about Python and the Python community. The most important point is the message that's sent back to aspiring speakers. Reviews that are based on the supposed character, or old talks, or how recognized you are or aren't, or on a guess as to which of multiple authors might be doing the presenting - all of those send a bad message. They make PyCon look insular and cliquey. If the committee of people is (or merely gives the impression of being) inwardly focused, the community and in the longer term perhaps the language itself will suffer through reduced diversity and through discouraging precisely the people who are animated enough and have the initiative and ambition to submit talks. Those are *exactly* the wrong folks to discourage. The obvious suggestion is to anonymize the review process. That's standard in mature conferences. It doesn't eliminate bias (in fact you *don't want* to eliminate bias - you need it to survive, you need it to assess quality), but it does reduce the opportunity for judgment based on the wrong things. When I say "wrong" I mean: if you're going to judge based on stuff that's not just the proposal content, then ask for a CV, or a speaking record, or whatever you intend to consider in the review process. Anonymizing conference reviewing has healthy effects. I've seen it up close in academic circles. It's like a breath of fresh air and the results are surprising. When they did it in the genetic algorithms world, all of a sudden really interesting talks were being accepted from all over the world and many very experienced researchers were having multiple talks rejected. That was unexpected, refreshing, and generally agreed to be a very healthy and embracing/welcoming move. If PyCon doesn't move to anonymizing reviews, then at least *try* not to base acceptance decisions on who a speaker is (or, worse, who it's presumed to be). If for some reason you have to, it's *perhaps* better not to tell the poor submitter that they're being rejected in part based on who they are or aren't. The CFP requests a talk proposal, and that's what should be reacted to in the review response, even if there's more to the story. Some of the comments above *might* be appropriate for a conference committee meeting, but not for the first (or only!) line of a review. OK, rant over :-) Regards to everyone & thanks for all the PyCon work. I know how much work it is, and that it's not easy. I hope to be able to make it to Atlanta. Terry Jones

Hello Terry, Terry Jones <terry@...> writes:
I am not part of the review board, and neither have I tried to submit a talk, and I'm not even sure this is the right mailing-list, but I found it interesting to read your personal report of your attempts to submit one. I also must say that, while I don't like popularity mechanisms myself, it's not very surprising to find them in the Python community since they exist everywhere else :) Thanks for the write-up. Antoine.

[Terry Jones]
The obvious suggestion is to anonymize the review process.
FWIW, that was tried and the people complained about that too. Who would you rather hear speak about the future of Python, Guido and someone else? About the state of Twisted, from someone on that team or from a user who read the Twisted book? About UnladedSwallow or AppEngine, someone on Google's team or someone who has played around with it for a while? Also, there are some folks like Alex Martelli whose talks I will seek out no matter what he's talking about (because it's always worthwhile). Likewise, it's not irrelevant if a speaker previously gave a talk that sucked. Surely, the review process has room for improvements and better balance but anonymizing is a step too far IMO. Raymond

On Tue, Nov 3, 2009 at 4:32 PM, Raymond Hettinger <python@rcn.com> wrote:
This is where a 2 stage review process helps. Stage one it to group all talks with the same topic and do a speaker validation process to make sure the speaker of a given topic is an expert in that area, or failing a known or verifiable expert proposing a talk in the area then all talks on that topic should be passed on to the anonymized process to be judged based on their proposal.
Also, there are some folks like Alex Martelli whose talks I will seek out no matter what he's talking about (because it's always worthwhile).
This may be true, but should have little to no bearing on the talk proposal process because if his talks are always good then his proposals are likly to be good as well and thus get approval based on that
Likewise, it's not irrelevant if a speaker previously gave a talk that sucked.
This is true as well. And should be considered in stage one of the process, but a single talk that sucked I would say is not a basis to reject someone outright. I would say two poor talks @pycon in the past three years warrants a negative vote, but a single bad talk, unless the voter attended that talk in person, isnt justification for a negative vote.

On Tue, Nov 3, 2009 at 15:32, Raymond Hettinger <python@rcn.com> wrote:
I agree. The year we went fully anonymous did not turn out as well as previous years. And I would also like to say this is off-topic for python-ideas. This would be better discussed on the pycon-pc list or directly at some other PyCon mailing list. -Brett

Raymond Hettinger writes:
That's what invited talks are for. Guido van Rossum or Alex Martelli, you invite them to give a keynote. But you can also salt the regular sessions with "invited" speakers. There's nothing that says that people can't suggest themselves for invitations.
Surely, the review process has room for improvements and better balance but anonymizing is a step too far IMO.
Anonymizing is the only way to get a reasonable balance between the very short-term view you are presenting, and the long-term view of encouraging new participants with good ideas and discouraging/warning old-timers whose ideas and views have gone stale, or even started to stink. Good proposals have a fairly high correlation with good talks; although you can't expect to win them all. You don't have to anonymize all the sessions/talks, either, but probably at least half should be refereed blind.

On Tue, Nov 3, 2009 at 10:08 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Invited speakers are invited, they don't self nominate. If you have suggestions to change this, please join the pycon mailing list(s).
I encourage you to participate in Pycon organization next year. This is not the proper forum. Jesse

On Tue, Nov 3, 2009 at 4:41 PM, Terry Jones <terry@jon.es> wrote: ...mega snip ...
Terry Jones
This is not the correct forum for this. If you have issues with the process, you can email me, as I am/was the acting committee chair for talks, Van Lindberg as the pycon chair, or the pycon-org, or pycon-pc lists. Jesse

Jesse Noller wrote:
I am not going to join the list but I have a couple of suggestions: If you are going judge speakers, do so openly; perhaps even tell poor speakers not to bother again. Proposals with multiple authors should then have blanks for 'intended speaker' and 'alternate speaker' so the right person can be judged. Editors of professional journals read reviews for acceptibility before sending them on to authors. Consider the same here. Terry Jan Reedy

[snip] It seems you are mistaken about the purpose of a conference (not only python, but any academic or professional gathering). Since the number of presenters is always much less than the number of people who listen, the goal of every good conference organizer should be to first look after the interest of the people who listen and only then look after the interest of the presenters. In other words, the number one goal is to have the audience enjoy the show. Once this is done, one can think about what is in the interest of the presenters. Whenever there is a conflict, the interest of the audience comes first. Making somebody well known in the python community by giving him a slot at pycon is not the number one goal of a conference. So if somebody is sad/unhappy/etc because he can not present, well, that's not a problem at all if the decision was made in the good faith that turning away the person and giving the slot to someone else will increase the enjoyment of the audience. I believe the cases you mentioned fall into this category. Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown

Hello Terry, Terry Jones <terry@...> writes:
I am not part of the review board, and neither have I tried to submit a talk, and I'm not even sure this is the right mailing-list, but I found it interesting to read your personal report of your attempts to submit one. I also must say that, while I don't like popularity mechanisms myself, it's not very surprising to find them in the Python community since they exist everywhere else :) Thanks for the write-up. Antoine.

[Terry Jones]
The obvious suggestion is to anonymize the review process.
FWIW, that was tried and the people complained about that too. Who would you rather hear speak about the future of Python, Guido and someone else? About the state of Twisted, from someone on that team or from a user who read the Twisted book? About UnladedSwallow or AppEngine, someone on Google's team or someone who has played around with it for a while? Also, there are some folks like Alex Martelli whose talks I will seek out no matter what he's talking about (because it's always worthwhile). Likewise, it's not irrelevant if a speaker previously gave a talk that sucked. Surely, the review process has room for improvements and better balance but anonymizing is a step too far IMO. Raymond

On Tue, Nov 3, 2009 at 4:32 PM, Raymond Hettinger <python@rcn.com> wrote:
This is where a 2 stage review process helps. Stage one it to group all talks with the same topic and do a speaker validation process to make sure the speaker of a given topic is an expert in that area, or failing a known or verifiable expert proposing a talk in the area then all talks on that topic should be passed on to the anonymized process to be judged based on their proposal.
Also, there are some folks like Alex Martelli whose talks I will seek out no matter what he's talking about (because it's always worthwhile).
This may be true, but should have little to no bearing on the talk proposal process because if his talks are always good then his proposals are likly to be good as well and thus get approval based on that
Likewise, it's not irrelevant if a speaker previously gave a talk that sucked.
This is true as well. And should be considered in stage one of the process, but a single talk that sucked I would say is not a basis to reject someone outright. I would say two poor talks @pycon in the past three years warrants a negative vote, but a single bad talk, unless the voter attended that talk in person, isnt justification for a negative vote.

On Tue, Nov 3, 2009 at 15:32, Raymond Hettinger <python@rcn.com> wrote:
I agree. The year we went fully anonymous did not turn out as well as previous years. And I would also like to say this is off-topic for python-ideas. This would be better discussed on the pycon-pc list or directly at some other PyCon mailing list. -Brett

Raymond Hettinger writes:
That's what invited talks are for. Guido van Rossum or Alex Martelli, you invite them to give a keynote. But you can also salt the regular sessions with "invited" speakers. There's nothing that says that people can't suggest themselves for invitations.
Surely, the review process has room for improvements and better balance but anonymizing is a step too far IMO.
Anonymizing is the only way to get a reasonable balance between the very short-term view you are presenting, and the long-term view of encouraging new participants with good ideas and discouraging/warning old-timers whose ideas and views have gone stale, or even started to stink. Good proposals have a fairly high correlation with good talks; although you can't expect to win them all. You don't have to anonymize all the sessions/talks, either, but probably at least half should be refereed blind.

On Tue, Nov 3, 2009 at 10:08 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Invited speakers are invited, they don't self nominate. If you have suggestions to change this, please join the pycon mailing list(s).
I encourage you to participate in Pycon organization next year. This is not the proper forum. Jesse

On Tue, Nov 3, 2009 at 4:41 PM, Terry Jones <terry@jon.es> wrote: ...mega snip ...
Terry Jones
This is not the correct forum for this. If you have issues with the process, you can email me, as I am/was the acting committee chair for talks, Van Lindberg as the pycon chair, or the pycon-org, or pycon-pc lists. Jesse

Jesse Noller wrote:
I am not going to join the list but I have a couple of suggestions: If you are going judge speakers, do so openly; perhaps even tell poor speakers not to bother again. Proposals with multiple authors should then have blanks for 'intended speaker' and 'alternate speaker' so the right person can be judged. Editors of professional journals read reviews for acceptibility before sending them on to authors. Consider the same here. Terry Jan Reedy

[snip] It seems you are mistaken about the purpose of a conference (not only python, but any academic or professional gathering). Since the number of presenters is always much less than the number of people who listen, the goal of every good conference organizer should be to first look after the interest of the people who listen and only then look after the interest of the presenters. In other words, the number one goal is to have the audience enjoy the show. Once this is done, one can think about what is in the interest of the presenters. Whenever there is a conflict, the interest of the audience comes first. Making somebody well known in the python community by giving him a slot at pycon is not the number one goal of a conference. So if somebody is sad/unhappy/etc because he can not present, well, that's not a problem at all if the decision was made in the good faith that turning away the person and giving the slot to someone else will increase the enjoyment of the audience. I believe the cases you mentioned fall into this category. Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown
participants (9)
-
Antoine Pitrou
-
Brett Cannon
-
Daniel Fetchinson
-
Dj Gilcrease
-
Jesse Noller
-
Raymond Hettinger
-
Stephen J. Turnbull
-
Terry Jones
-
Terry Reedy