[Neuroimaging] ANN: openneuro-py, a new app for downloading OpenNeuro datasets

[Christopher Markiewicz]

Hi Richard,

Yarik beat me to most of this, but to address this issue:

> *   the sync between OpenNeuro and GitHub doesn't seem to work reliably, so we were having trouble retrieving the latest revisions of datasets using datalad

This is definitely a work in progress, but a lot of effort has gone into resolving the synchronization issues, and the situation is improving rapidly. If you do find cases where the website/S3/GitHub are out of sync with each other, could you check the issues linked in https://github.com/OpenNeuroOrg/openneuro/issues/1895 and make sure we know about it and can track it?

Also, 

> *   as far as I understand, there's no way to download a specific revision (version number) of a dataset using datalad

Note that snapshots are all git tags. The CLI equivalent of Yarik's Python code is just `datalad install ...; git -C <dataset> checkout <tag>`.

---

I don't want to argue writing/using the tool. It looks useful, and there are definitely cases where it's needed at this point. I do want to warn that it may end up being a short-term solution. OpenNeuro has made no commitment to maintain a stable web API; while there's not going to be breakage for its own sake, you do have a potential moving target as the needs of the platform change. The stable interface that we're trying to work toward full support for is datalad, which has well-established storage and transport models.

Anyway, apologies for redirecting this thread to being largely about datalad.

Best,
Chris

________________________________________
From: Neuroimaging <neuroimaging-bounces+markiewicz=stanford.edu at python.org> on behalf of Richard Höchenberger <richard.hoechenberger at gmail.com>
Sent: Tuesday, December 15, 2020 11:50 AM
To: Neuroimaging analysis in Python
Subject: Re: [Neuroimaging] ANN: openneuro-py, a new app for downloading OpenNeuro datasets

Hello, just to briefly comment on this one:

On Tue, Dec 15, 2020 at 4:27 PM Christopher Markiewicz <markiewicz at stanford.edu<mailto:markiewicz at stanford.edu>> wrote:
Hi all,

FWIW almost all public datasets have been pushed to GitHub and can be accessed via datalad (exceptions being tracked on these issues: https://github.com/OpenNeuroOrg/openneuro/issues/1741 and https://github.com/OpenNeuroOrg/openneuro/issues/1743).

    datalad install https://github.com/OpenNeuroDatasets/ds00WXYZ.git


I specifically wanted to avoid using datalad for several reasons:

  *   I've seen people struggle to install (esp. on macOS) and use it (including myself)
  *   the sync between OpenNeuro and GitHub doesn't seem to work reliably, so we were having trouble retrieving the latest revisions of datasets using datalad
  *   as far as I understand, there's no way to download a specific revision (version number) of a dataset using datalad

openneuro-py attempts to do away with these issues.

Best wishes,

    Richard