At both SciPy conferences, Eric emphasized the need for a peer-review process for potential contributions to SciPy. I recalled that the Boost community established a formal review process for contributions to their set of libraries, and it seems to work well for them. I got bored of writing a history paper, so I looked at their process and wrote up a proposal for SciPy (liberally using parts of the Boost documents where noted). Please comment on, add to, delete from, and otherwise contribute to the following proposal. You can email me personally as well as respond to the list. As he announced at SciPy '03, Chad Netzer has a wavelets module he would like to submit. If we can hammer out some consensus (enough for a test run at least), then Chad can be our first sacrif^Wclient. I am sure he is as thrilled to learn of this possibility as I am to raise it. The text is included below. HTML and text versions can also be found in the following URLs: http://www.ugcs.caltech.edu/~kern/scipy-review.html http://www.ugcs.caltech.edu/~kern/scipy-review.txt A Proposal for a SciPy Module Submission Review Process ======================================================= :Author: Robert Kern (parts by the Boost_ group and others__) :Version: 0.0 :Date: 2003-09-20 The following proposal is inspired by (and indeed partly outright lifted from) the `Boost Formal Review Process`_ and `Submission Process`_. It is written using ReStructuredText_. .. _Boost: http://www.boost.org __ Acknowledgements_ .. _Boost Formal Review Process: http://www.boost.org/more/formal_review_process.htm .. _Submission Process: http://www.boost.org/more/submission_process.htm .. _ReStructuredText : http://docutils.sourceforge.net Rationale --------- 1. The SciPy package requires a peer-review process to examine new module submissions. The expertise of the CVS maintainer is limited, and the number of fields for which one could write a SciPy module is large. 2. A formal procedure for the review process makes it easier for newcomers to know exactly what they have to do to get their contribution into SciPy. Furthermore, a formal procedure makes it easier for Enthought to parcel out the work to non-Enthought people and even to transition SciPy to a wholly community-driven project if that ever becomes a goal. 3. The Boost project had a similar problem, found a solution that worked for them, and wrote it up all pretty, so why not steal it? [XXX - or something to this effect] Organization ------------ Mailing List ~~~~~~~~~~~~ Discussion of libraries under formal review shall take place on scipy-dev@scipy.org . Actual reviews of a candidate module (see section `Reviews`_) during its formal review period should begin their Subject header with [REVIEW - <name of module>] for the convenience of the Review Manager. Reviewers who do not wish to subscribe to scipy-dev@scipy.org may use the GMane_ WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows posting for this list, either.] .. _GMane: http://www.gmane.org .. _WWW: http://news.gmane.org/gmane.comp.python.scientific.devel .. _Usenet: nntp://news.gmane.org/gmane.comp.python.scientific.devel Review Wizard ~~~~~~~~~~~~~ (The Review Wizard/Manager sections are only slightly changed from `Boost Formal Review Process`_.) The Review Wizard: * Maintains a list of review manager volunteers. * When a formal review is requested for a module: - Assigns a review manager and suggests a schedule. - Finalizes the schedule, once the review manager verifies the module is actually ready for review. - Resolves schedule slips or other issues with review managers and submitters. * Maintains a schedule of both past and pending reviews. * Resolves questions from review managers and module submitters, who sometimes want a third opinion on questions such as "Should we extend the review period because ...?" * Monitors the general review process, and makes minor adjustments as needed, or queries the list about possible major adjustments. .. _Boost Formal Review Process: http://www.boost.org/more/formal_review_process.htm Review Manager ~~~~~~~~~~~~~~ The Review Manager: * Checks the submission to make sure it really is complete enough to warrant formal review. If necessary, work with the submitter to verify the code compiles and runs correctly on several platforms. * Finalizes the schedule with the Review Wizard and the submitter. * Posts a notice of the review schedule on the appropriate mailing list(s). - The notice should include a brief description of the module and what it does, to let readers know if the module is one they are interested in reviewing. - If the module is known to fail on certain platforms, please mention them in the review notice so reviewers on those platforms won't waste time diagnosing known problems (or alternatively, will test and try to resolve the failures). * Inspects the SciPy package for modules which may interact with the new submission. These potential interactions should be pointed out in the review announcement, and the author(s) of these modules should be privately notified and urged to participate in the review. * Urges people to do reviews if they aren't forthcoming. * Follows review discussions regarding the modules, moderating or answering questions as needed. * Decides if there is consensus to accept the modules, and if there are any conditions attached. * Posts a message to the appropriate mailing lists informing people of the review results. In other words, it is the Review Manager's responsibility to make sure the review process works smoothly. The Process for a Module Submitter ---------------------------------- Determine Interest ~~~~~~~~~~~~~~~~~~ Post a message to scipy-dev@scipy.org to see if there is interest in your module. Describe what your module does and perhaps include a small snippet of code that shows how one uses it. Preliminary Submission ~~~~~~~~~~~~~~~~~~~~~~ Put your (possibly unfinished) module on a website and post its location to scipy-dev@scipy.org . Contact the maintainers of www.scipy.org for a Zope account if you need web space for your module. Submissions should follow the Code Submission Guidelines [XXX - this should point to something useful. For now, look at the `Code Formatting Guidelines`_ and the `Testing Guidelines`_.] .. _Code Formatting Guidelines: http://www.scipy.org/site_content/tutorials/formatting_guidelines .. _Testing Guidelines: http://www.scipy.org/site_content/tutorials/testing_guidelines Refinement ~~~~~~~~~~ Discuss the module in scipy-dev@scipy.org . Fix bugs; add functionality; change the interface; write docs/unit tests; do whatever comes out of the discussion as necessary or beneficial to including the module in SciPy. Update the preliminary submission until it is ready for formal submission. Submission for Review ~~~~~~~~~~~~~~~~~~~~~ Package your module submission in a ZIP file or gzipped tar file such that it can be unpacked in the scipy CVS directory [XXX - what if you need to patch the setup.py file or scipy_distutils files? Diff file?]. Contact the Review Wizard. He will assign a Review Manager and create a www.scipy.org Zope account for you. Place your submission in [XXX] on the www.scipy.org website. The Review Manager will kick off the reviews on scipy-dev@scipy.org . Reviews ------- (Again, taken from `Boost Formal Review Process`_). Reviews may be submitted to either the list as a whole or the Review Manager privately. Your comments may be brief or lengthy, but basically the Review Manager needs your evaluation of the module. If you identify problems along the way, please note if they are minor, serious, or showstoppers. Here are some questions you might want to answer in your review: * What is your evaluation of the design? * What is your evaluation of the implementation? * What is your evaluation of the documentation? * What is your evaluation of the unit-test coverage? * What is your evaluation of the potential usefulness of the module? * Did you try to use the module? On what platform? Did you have any problems? * How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? * Are you knowledgeable about the problem domain? Are you knowledgeable about the programming techniques used? [E.g. perhaps you know nothing about what the underlying FORTRAN code does, but you are an expert in wrapping code with f2py. Both areas of knowledge are useful for a review.] And finally, every review should answer this question: * Do you think the module should be accepted into the SciPy package? Be sure to say this explicitly so that your other comments don't obscure your overall opinion. The module should be stable during the review period (making exception for serious bug fixes). Modifications suggested during review and accepted by the submitter should be noted on the mailing list so reviewers can make an informed decision. At the end of the review period, the Review Manager will make the final decision to accept or reject the module for inclusion into SciPy. He will post this result to the mailing list along with a rationale and any suggestions or conditions that must be met before the module can be included (or resubmitted for review, as the case may be). Acknowledgements ---------------- I would like to thank the Boost community for their well thought-out review process and the words describing said process. [XXX - Eric, please check if this is OK with them] Eric Jones, [XXX - Your Names Here] provided helpful comments and contributions to this document. [XXX - Kern's Notes] -------------------- The Review Wizard/Manager roles will probably be taken by Enthought people or their designees at least at the start, mostly because they would have to do similar tasks without a formal review process, so who else would volunteer? Note the XXX's. They need content or answers. Boost adds the libraries up for review into a CVS sandbox. We may want to do the same. The process as a whole feels a bit heavyweight to me, but there are some benefits that flow from that. Some nips and tucks couldn't hurt, though. -- Robert Kern kern@caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
On Sat, 20 Sep 2003, Robert Kern wrote:
A Proposal for a SciPy Module Submission Review Process =======================================================
:Author: Robert Kern (parts by the Boost_ group and others__) :Version: 0.0 :Date: 2003-09-20
The following proposal is inspired by (and indeed partly outright lifted from) the `Boost Formal Review Process`_ and `Submission Process`_. It is written using ReStructuredText_.
Did you check out PEP-2? I think there are several things that the Boost document does not cover: maintaining a module, for one instance.
Rationale ---------
1. The SciPy package requires a peer-review process to examine new module submissions. The expertise of the CVS maintainer is limited, and the number of fields for which one could write a SciPy module is large.
2. A formal procedure for the review process makes it easier for newcomers to know exactly what they have to do to get their contribution into SciPy. Furthermore, a formal procedure makes it easier for Enthought to parcel out the work to non-Enthought people and even to transition SciPy to a wholly community-driven project if that ever becomes a goal.
Agreed.
3. The Boost project had a similar problem, found a solution that worked for them, and wrote it up all pretty, so why not steal it? [XXX - or something to this effect]
Hmm, I guess there are several projects with the same problem. Python, for instance. I would leave this item out and mention Boost, say, in acknowledgements (somehow so many references to Boost were annoying to me while reading this document;-).
Organization ------------
Mailing List ~~~~~~~~~~~~
Discussion of libraries under formal review shall take place on scipy-dev@scipy.org . Actual reviews of a candidate module (see section `Reviews`_) during its formal review period should begin their Subject header with [REVIEW - <name of module>] for the convenience of the Review Manager.
Reviewers who do not wish to subscribe to scipy-dev@scipy.org may use the GMane_ WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows posting for this list, either.]
AFAIK, only members can post to scipy-dev mailing list. <snip>
The Process for a Module Submitter ----------------------------------
Determine Interest ~~~~~~~~~~~~~~~~~~
Post a message to scipy-dev@scipy.org to see if there is interest in your module. Describe what your module does and perhaps include a small snippet of code that shows how one uses it.
Preliminary Submission ~~~~~~~~~~~~~~~~~~~~~~
Put your (possibly unfinished) module on a website and post its location to scipy-dev@scipy.org . Contact the maintainers of www.scipy.org for a Zope account if you need web space for your module.
Submissions should follow the Code Submission Guidelines [XXX - this should point to something useful. For now, look at the `Code Formatting Guidelines`_ and the `Testing Guidelines`_.]
See also PEP-7 and PEP-8. Along the lines in PEP-2, submission should include a rather complete documentation in order to get a better picture of the module when it will be finished. Such a document may contain also ideas/suggestions on future extensions of the module by proposing, say, function signatures. Such functions are allowed to be implemented even after the module is accepted to Scipy package. Since Scipy should provide a solid base for scientific work, all modules should have a mathematical background documented, most importantly, the formal definitions of used concepts. This would allow verifying the correctness of the results as well as avoids misusing the provided tools by the end users who might know/use slightly different definitions for some mathematical concepts.
Refinement ~~~~~~~~~~
Discuss the module in scipy-dev@scipy.org . Fix bugs; add functionality; change the interface; write docs/unit tests; do whatever comes out of the discussion as necessary or beneficial to including the module in SciPy. Update the preliminary submission until it is ready for formal submission.
Submission for Review ~~~~~~~~~~~~~~~~~~~~~
Package your module submission in a ZIP file or gzipped tar file such that it can be unpacked in the scipy CVS directory [XXX - what if you need to patch the setup.py file or scipy_distutils files? Diff file?].
I think we can work out a setup where no modifications to setup.py files are needed. I am thinking of adding (or using existing pre__init__.py file) a meta file into each module directory in scipy/Lib and scipy/sandbox that contains the following information about the module: - scipy level information - is it standalone or to be installed under scipy, etc. - state of the module (final,beta,...,unstable,ignore) - documentation - platform or software dependencies (setup.py would ignore the module when not running on the required platform or if some software is not installed) - etc. Then setup.py will scan directories and decides whether or not to include a module to a building/installation/distribution process based on the content of pre__init__.py file in these directories. Such a setup should be as simple to use as possible: adding a new module would require two steps: 1) unpack module to the sandbox directory 2) create a pre__init__.py file to the module directory (unless submitter already had done that) Pearu
Pearu Peterson wrote:
On Sat, 20 Sep 2003, Robert Kern wrote:
A Proposal for a SciPy Module Submission Review Process =======================================================
:Author: Robert Kern (parts by the Boost_ group and others__) :Version: 0.0 :Date: 2003-09-20
The following proposal is inspired by (and indeed partly outright lifted from) the `Boost Formal Review Process`_ and `Submission Process`_. It is written using ReStructuredText_.
Did you check out PEP-2? I think there are several things that the Boost document does not cover: maintaining a module, for one instance.
Not until now, no. But you're right, it has good ideas to steal. The integrator/maintainer concepts are quite useful. And yes, maintenance needs to be addressed though I think it should be fully covered in a separate SciPy Development Guide or something and briefly summarized here. [snip]
3. The Boost project had a similar problem, found a solution that worked for them, and wrote it up all pretty, so why not steal it? [XXX - or something to this effect]
Hmm, I guess there are several projects with the same problem. Python, for instance.
You're right, and I will look at the PEPs you suggest. Boost was just the most salient project in my head when I started thinking about this.
I would leave this item out and mention Boost, say, in acknowledgements (somehow so many references to Boost were annoying to me while reading this document;-).
Sorry. When I commit larceny on so comprehensive a scale, I feel guilty and try to give advance warning. :-)
Organization ------------
Mailing List ~~~~~~~~~~~~
Discussion of libraries under formal review shall take place on scipy-dev@scipy.org . Actual reviews of a candidate module (see section `Reviews`_) during its formal review period should begin their Subject header with [REVIEW - <name of module>] for the convenience of the Review Manager.
Reviewers who do not wish to subscribe to scipy-dev@scipy.org may use the GMane_ WWW_ and Usenet_ interfaces to the mailing list. The WWW_ interface does not allow posting. [XXX - to be honest, I don't know if the Usenet_ one allows posting for this list, either.]
AFAIK, only members can post to scipy-dev mailing list.
I am posting this via GMane, so it must work.
<snip>
The Process for a Module Submitter ----------------------------------
Determine Interest ~~~~~~~~~~~~~~~~~~
Post a message to scipy-dev@scipy.org to see if there is interest in your module. Describe what your module does and perhaps include a small snippet of code that shows how one uses it.
Preliminary Submission ~~~~~~~~~~~~~~~~~~~~~~
Put your (possibly unfinished) module on a website and post its location to scipy-dev@scipy.org . Contact the maintainers of www.scipy.org for a Zope account if you need web space for your module.
Submissions should follow the Code Submission Guidelines [XXX - this should point to something useful. For now, look at the `Code Formatting Guidelines`_ and the `Testing Guidelines`_.]
See also PEP-7 and PEP-8.
Along the lines in PEP-2, submission should include a rather complete documentation in order to get a better picture of the module when it will be finished. Such a document may contain also ideas/suggestions on future extensions of the module by proposing, say, function signatures. Such functions are allowed to be implemented even after the module is accepted to Scipy package.
Since Scipy should provide a solid base for scientific work, all modules should have a mathematical background documented, most importantly, the formal definitions of used concepts. This would allow verifying the correctness of the results as well as avoids misusing the provided tools by the end users who might know/use slightly different definitions for some mathematical concepts.
Agreed. As alluded to above, I think all of these requirements should be wrapped up in a separate SciPy Development Guide since the information is useful to maintainers as well as potential contributors and because it is likely to be longer than this document is already. [snip]
Such a setup should be as simple to use as possible: adding a new module would require two steps: 1) unpack module to the sandbox directory 2) create a pre__init__.py file to the module directory (unless submitter already had done that)
Nifty! +1. However, we still may want to deal with the issue of modules needing to modify, say, scipy_core/scipy_distutils/system_info.py .
Pearu
-- Robert Kern kern@ugcs.caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
Hi Robert,
"RK" == Robert Kern <kern@ugcs.caltech.edu> writes:
RK> At both SciPy conferences, Eric emphasized the need for a RK> peer-review process for potential contributions to SciPy. I RK> recalled that the Boost community established a formal review RK> process for contributions to their set of libraries, and it RK> seems to work well for them. I got bored of writing a history RK> paper, so I looked at their process and wrote up a proposal RK> for SciPy (liberally using parts of the Boost documents where RK> noted). Nice! Just one suggestion. You probably need to mention that the code must be released under the BSD licence in order to be considered. I guess the MIT and the Python license are also fine but I don't know what exactly the official stance of the SciPy community is on this. cheers, prabhu
On Tue, Sep 23, 2003 at 10:41:25PM +0530, Prabhu Ramachandran wrote:
Hi Robert,
"RK" == Robert Kern <kern@ugcs.caltech.edu> writes:
RK> At both SciPy conferences, Eric emphasized the need for a RK> peer-review process for potential contributions to SciPy. I RK> recalled that the Boost community established a formal review RK> process for contributions to their set of libraries, and it RK> seems to work well for them. I got bored of writing a history RK> paper, so I looked at their process and wrote up a proposal RK> for SciPy (liberally using parts of the Boost documents where RK> noted).
Nice! Just one suggestion. You probably need to mention that the code must be released under the BSD licence in order to be considered. I guess the MIT and the Python license are also fine but I don't know what exactly the official stance of the SciPy community is on this.
To address this issue and other requirements placed on new modules, I have written up a SciPy Development Guide. Since I made most of it up myself, I'm waiting for Eric to review it before I release it into the wild. The Guide will be included by reference in the review process document.
cheers, prabhu
-- Robert Kern kern@ugcs.caltech.edu "In the fields of Hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
participants (5)
-
Pearu Peterson -
Prabhu Ramachandran -
Robert Kern -
Robert Kern -
Robert Kern