[Neuroimaging] Technical details managing Python versions and packages.

[Satrajit Ghosh]

hi folks,

on the note of slightly more general package management.

for our most recent cluster we have switched to a project specific
versioning system with modulefiles. each project creates its own
environment (not just python but across other tools as well). for python,
most of our cluster users use miniconda/anaconda for their environments and
it has worked really well. particularly, when they need to install other
non-python dependencies that conda folks have already built. each project
can maintain or update their stack themselves.

unlike personal computers or the cloud, on our cluster we need simultaneous
access to multiple versions of packages and libraries and we cannot run
docker (requires uid==0 privileges since the kernels we run have no
namespace support). so traditional packaging solutions simply don't work.
modulefiles and lmod is what TACC uses to support packages across their
userbase, and for us following that model got us up and running relatively
quickly. further, we are able to share such packaging across local
institutions co-located in a central cluster computing facility. setting up
openstack was not possible because of GPU needs. we also allow running vm's
with vagrant, where people can use more traditional environments with
package management. however, GPU passthroughs are still not clean. we
encourage users to do their own testing.

this project offers an alternative cross-platform solution to creating
certain software stacks.

https://hashdist.github.io/

i would love to hear if there are other solutions that can provide
user/group-level software install in an HPC setting. we do need support for
access to hardware such as GPUs and these solutions have to be usable
within queuing systems.

cheers,

satra

On Mon, Aug 3, 2015 at 6:06 PM, JB Poline <jbpoline at gmail.com> wrote:

>
>
> On Mon, Aug 3, 2015 at 2:10 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>
>> On Mon, Aug 3, 2015 at 5:49 PM, JB Poline <jbpoline at gmail.com> wrote:
>> > That's a very useful page to me -
>> > A quick possible addition : why / when use the  --user option ? (and get
>> > packages in .local/... )
>>
>> I personally don't use --user very often, just because I have got used
>> to starting a new virtualenv whenever I do something.   For example,
>> if I am working on dipy I might do:
>>
>> mkvirtualenv dipy-work
>> pip install -r requirements.txt
>>
>> This takes a few seconds on my machine because the relevant wheels are
>> built and cached.
>>
>
> Looks like this would be a nice little addition to your explanations
>
>
>>
>> > I also wonder if examples of trouble shooting or install that raised
>> > difficulties could be added there in a "use cases" section : by reading
>> how
>> > some issues where solved I'm sure I could learn a bit more how to debug
>> > install that go wrong.
>>
>> How about raising an Issue on that repo when you run into trouble, and
>> we can try and add something useful on solving the problem?
>>
>> https://github.com/matthew-brett/pydagogue/issues
>>
>> I'll add something at the bottom of the page.
>>
>
>
> ok - will do when I run into trouble (so probably often and soon)
> see ya
> JB
>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20150805/afa8bef9/attachment.html>