[Neuroimaging] [DIPY] Setting up a platform for offline end-to-end quality assurance for DIPY

[Chris Filo Gorgolewski]

Have a look at waht we are doing for Nipype on CircleCI (on the free open
source tier):

https://github.com/nipy/nipype/blob/master/circle.yml
https://circleci.com/gh/nipy/nipype

All of the workflows we run for tests take over 3h to finish. Similar set
up is implemented in nilearn project.

Best,
Chris

On Thu, Mar 3, 2016 at 10:29 AM, Ariel Rokem <arokem at gmail.com> wrote:

> Hi Eleftherios,
>
> I have resources to run this kind of thing on AWS, or some other cloud
> provider. I see many advantages to doing this on the cloud and using
> something like docker for deployment (e.g., portability and reproducibility
> in other people's hands, as well as relatively easy scaling in ours). Data
> can then also consistently be pulled from the HCP S3 buckets (see for
> example the beginning of the notebook here:
> https://github.com/arokem/end-to-end/blob/master/end-to-end.ipynb). Once
> we have automated all that, it will also be relatively easy to transfer
> these ideas to the other use-cases you mentioned.
>
> But we'd need to do some math to see how much this would actually cost. Do
> you have a sense of the requirements? For example, how often would you want
> to run the pipeline? Every time a PR happens? That's happening quite often
> these days ;-) I don't believe we need a really large machine to run
> persistently. We might want a small machine running persistently,
> monitoring github for us, and then waking up the big beast when there's a
> lot of work to do. That might reduce costs.
>
> Cheers,
>
> Ariel
>
> On Thu, Mar 3, 2016 at 8:24 AM, Eleftherios Garyfallidis <
> garyfallidis at gmail.com> wrote:
>
>> Dear Matthew, Maxime, Ariel and all,
>>
>> Mr. Dumont and I have started creating some workflows which can be run by
>> the command line. These are made to work with large real datasets.
>>
>> I think it would be great if we could use a different type of testing
>> from what we were using right now. Most of the testing we use is actually
>> fast testing of functions and we should definitely continue having that.
>>
>> But I think we need also an end-to-end offline testing where we actually
>> test with big whole brain datasets and then we can collect some automatic
>> quality assurance reports. In that way we cover most of unexpected issues.
>>
>> Now, the problem with having such a platform is that it needs computing
>> power and some disk space. It may need a descent computer to run for 24
>> hours for example and let's say around 100 GBytes of free disk space. Then
>> it will also need to send some automated reports to say that is all good or
>> not.
>>
>> Ariel has suggested  to use the cloud and docker but I am afraid that it
>> will be too expensive for our pockets right now except if someone can
>> donate to the project.
>>
>> An alternative idea would be to go gradually and setup one of the
>> computers in Sherbrooke or in Berkeley or in Seattle to do such a job. I
>> think this QA should run once/twice a week rather than every day.
>>
>> Now there are other platforms that need to run relatively frequently. One
>> is the examples for the documentation and then there is Omar's validation
>> framework which actually needs a large cluster. We can deal with those at a
>> later stage.
>>
>> The easiest way forward with the workflows that I see right now is that
>> Mr. Dumont adds a script in dipy/tools that will run all the workflows as
>> we do with make_examples.py that run all the examples. We first try this
>> platform in Sherbrooke and then we need to figure out a way to send
>> automated reports to the core developers or to berkeley builders and so on.
>> Maybe sending a PDF or HTML of the output screenshots would be also a
>> good idea.
>>
>> Let me know what you think.
>>
>> Cheers,
>> Eleftherios
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Neuroimaging mailing list
>> Neuroimaging at python.org
>> https://mail.python.org/mailman/listinfo/neuroimaging
>>
>>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20160303/8a6634ac/attachment.html>