[Inpycon] PyCon proposal review process

[Jaseem Abid]

On Mon, Jun 2, 2014 at 3:12 AM, me kracekumar <me at kracekumar.com> wrote:

> Hi
>
>  We received 143 proposals and 30 proposals on last day.
>

First of all, congratulations!

I have some experience in reviewing talk proposals and I would like to
volunteer and help reviewing a few proposals, which I can or know about.

I'd like to clarify a few questions before getting started.

1. Is the review process public?
2. Is there someone who can override suggestions from the review team?
3. Is this how it was done last year too?


Review process:
>
>  Review process will happen in two phases. During phase1 of review, each
> reviewer will give +1, 0, -1 for each proposal. After the phase 1, any talk
> with no negatives will be accepted and with >3 negatives is rejected.
> Remaining talks will move to phase 2. In Phase 2, reviews will happens in
> group. After phase 2 final list will be announced. Reviewer is free to
> communicate to the proposer through funnel.
>

I'd like to propose a *slightly* different model. I don't think a single
number from [-1, 0, +1] will be good enough.

Julia Evans recently wrote an interesting article[1] on the point. Rather
than counting votes of +1 and -1, it could be more useful to give grades
for talks and make the reviewer advocate for it.

Here is the important section from the blog:

>> Talk review: Letter grades

>>[...] Each of us assigned a proposal a letter grade (A-D), from “I will
fight for this talk”, to “I will fight for this talk to be rejected”. We
then looked at the highest & lowest grade for each talk and binned them
into AA, AB, AC, AD, BB, BC, BD, CC, CD, DD. We rejected any talk that
didn’t have an A.

So in effect, any talk which got a D from any of the reviewers need to be
checked again. For example, If I have attended a previous talk from the
speaker which went really bad, I'd vote a D. Same goes for A too. An A from
a senior experienced person can be given more weightage.



> What does votes mean ?
>
> +1 -- The topic sounds good and the proposal is solid. This vote means
> reviewer is willing to put his/her reputation on the line in favour of the
> proposal.
>
> 0 -- This topic is good and the proposal is solid. The speaker is capable
> of correcting any deficiencies, and significant number of attendees want it.
>
> -1 -- This talk or its proposal has significant problems and would be
> worse for having it.
>
> Some criteria for evaluation are
>
> - Will I attend the talk if it is selected ?
> - Are there a significant group of attendees who want to attend the talk?
> - Do author know enough about the topic?
> - Is the proposal too ambitious for the time allotted?
>
> We will be publishing this as blog post soon.
>

This obviously require more discussions.

Comments appreciated.

1. http://jvns.ca/blog/2014/05/28/anonymous-talk-submission-equals-amazing/

-- 
Regards,

Jaseem Abid
github.com/jaseemabid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/inpycon/attachments/20140602/2cf49c53/attachment-0001.html>