[Neuroimaging] parallel computation of bundle_distances_mam/mdf ?
Emanuele Olivetti
olivetti at fbk.eu
Thu Dec 15 11:02:22 EST 2016
Hi Ariel,
I put the code in a previous message and did not attach it my last message,
on purpose. Now I am attaching it here, for your convenience. Notice that
is the same test.pyx/setup.py you succeeded to compile on your recent OSX,
wihtout OpenMP support. You can just import that module and invoke
test.test_func() to see the output I mentioned:
python -c "import test; test.test_func()"
Anyway, what I now believe is even more interesting, is to test something
simpler:
https://gist.github.com/romeroyonatan/6a380e189cee4b1290700cbc5259c837
That is a tiny piece of C code that does basically the same of test.pyx but
without the extra complexity of Cython. I tried it on GNU/Linux
(gcc+libgomp) and it works as expcted. Then I tried it on Paolo's laptop
(OSX 10.11.5) and it fails both with CLang and with Anaconda's
gcc4.8+libgomp (= "conda install gcc"). So this example narrows down the
problem to OpenMP only on OSX - no Python, Cython or else. Could you please
try compile and run it? As indicated there, you just need to:
gcc -fopenmp hello-openmp.c
./a.out
I am very surprised that such a minimal example fails to compile on OSX
with Anaconda's gcc4.8+libgomp. If someone has an explanation of such
issue, I'd really like to know more about it.
Best,
Emanuele
On Thu, Dec 15, 2016 at 4:09 PM, Ariel Rokem <arokem at gmail.com> wrote:
> Emanule,
>
> On Thu, Dec 15, 2016 at 12:41 AM, Emanuele Olivetti <olivetti at fbk.eu>
> wrote:
>
>> Thank you to all of you for the many details that help to clear up the
>> picture.
>>
>> It is my understanding that even the last version of OSX, i.e. v10.12.*,
>> does not support libomp within the Apple's Xcode development framework. At
>> the same time, future version should support it, at some point. Honestly,
>> this does not seem an ideal scenario, because not many OSX users chase the
>> latest versions of the opeating system, while everyone has multiple cores
>> since many years. For the same reason, I doubt that contacting the Xcode
>> side may bring us closer to the solution.
>> To Ariel: if you want to test that OpemMP works, once you build test.so,
>> just import it in the Python console and execute test.test_func(). You
>> should get this:
>>
>
> Was there supposed to be a file attached?
>
> Thanks!
>
> Ariel
>
>
>> ---
>> >>> import test
>> >>> test.test_func()
>> Thread ID: 2
>> Thread ID: 8
>> Thread ID: 7
>> Thread ID: 3
>> Thread ID: 4
>> Thread ID: 5
>> Thread ID: 6
>> Thread ID: 0
>> Thread ID: 1
>> Thread ID: 9
>> ---
>> If you do not enable the OpenMP support (just comment out the two related
>> lines in setup.py and re-build), the output will differ and only one thread
>> will show up:
>> ---
>> >>> import test
>> >>> test.test_func()
>> Thread ID: 0
>> ---
>>
>> Anyway, what surprises me most is the failure of Anaconda's gcc+libgomp
>> on OSX. In principle that toolchain is exactly the same as on GNU/Linux, so
>> I see no reason for it fail. I'll try to dig more into this one.
>>
>> To Eleftherios: I agree with you that OpenMP is the best technical
>> solution, because the use of threads - which shares memory - is definitely
>> more preferable than multiprocessing - which replicates data in each
>> process. Anyway, the limitations I see are still there:
>> 1) OpenMP support seems problematic on non-GNU/Linux systems. By the way,
>> what is the situation on Windows? Here we don't have such system so we
>> couldn't try.
>> 2) OpemMP works only with Cython code. This means that no Python part of
>> DiPy can use parallel execution. This seems a bit too excessive to me,
>> because I expect that not all parts of DiPy will be re-written in Cython,
>> in the foreseeable future.
>>
>> So, again, I advocate for some use of the standard multiprocessing module
>> :) as an temporary step when OpenMP cannot be used.
>>
>> Best,
>>
>> Emanuele
>>
>>
>> On Wed, Dec 14, 2016 at 10:04 PM, Stephan Meesters <
>> stephan.meesters at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The instructions at http://nipy.org/dipy/instal
>>> lation.html#openmp-with-osx are outdated since the clang-omp formula
>>> does not exist anymore.
>>>
>>> With the release of Clang 3.8.0 (08 Mar 2016), OpenMP 3.1 support is
>>> enabled in Clang by default. You will need the -fopenmp=libomp flag while
>>> building.
>>>
>>> I started a while back on a DIPY homebrew formula to allow installation
>>> via "brew install dipy", see https://github.com/Homebrew/ho
>>> mebrew-python/pull/310. However it was based around the deprecated
>>> clang-omp and I didn't get around to fix it. Might look into it soon if I
>>> find some spare time to tweak the formula.
>>>
>>> Regards,
>>> Stephan
>>>
>>> 2016-12-14 17:14 GMT+01:00 Samuel St-Jean <stjeansam at gmail.com>:
>>>
>>>> That also depends on which version of clang ships by default with osx,
>>>> in which case you have to play around with it to get a new one. I think it
>>>> starts with clang 3.7 to have openmp support (I only ever tried Mac osx
>>>> 10.9, so anyone more experienced can chime in), but everything older has to
>>>> go through the hombebrew gcc install and company. Might be worhtwhile to
>>>> check if openmp support is out of the box now, and starting on which mac
>>>> osx versions, since older ones could be problematic for first time
>>>> installs.
>>>>
>>>> I also have to admit I have no idea how old is old in the mac world, so
>>>> maybe 10.9 is already phased out by now, but it was a hard and time
>>>> consuming building around stuff with homebrew experience (and again, first
>>>> time I used a mac, so, I guess the average user would also have some
>>>> issues).
>>>>
>>>> Samuel
>>>>
>>>> 2016-12-14 16:51 GMT+01:00 Eleftherios Garyfallidis <elef at indiana.edu>:
>>>>
>>>>>
>>>>> Hi Emanuele,
>>>>>
>>>>> My understanding is that openmp was only temporarily not available
>>>>> when clang replaced gcc in osx.
>>>>>
>>>>> So, I would suggest to go ahead with openmp. Any current installation
>>>>> issues are only temporarily for osx.
>>>>> Openmp gives us a lot of capability to play with shared memory and it
>>>>> is a standard that will be around
>>>>> for very long time. Also, the great integration in cython makes the
>>>>> algorithms really easy to read.
>>>>> So, especially for this project my recommendation is to use openmp
>>>>> rather than multiprocessing. All the way! :)
>>>>>
>>>>> I am CC'ing Stephan who wrote the instructions for osx. I am sure he
>>>>> can help you with this. I would also suggest
>>>>> to check if xcode provides any new guis for enabling openmp. I
>>>>> remember there was something for that.
>>>>>
>>>>> Laterz!
>>>>> Eleftherios
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 14, 2016 at 6:29 AM Emanuele Olivetti <olivetti at fbk.eu>
>>>>> wrote:
>>>>>
>>>>>> Hi Eleftherios,
>>>>>>
>>>>>> Thank you for pointing me to the MDF example. From what I see the
>>>>>> Cython syntax is not complex, which is good.
>>>>>>
>>>>>> My only concern is the availability of OpenMP in the systems where
>>>>>> DiPy is used. On a reasonably recent GNU/Linux machine it seems
>>>>>> straightforward to have libgomp and the proper version of gcc. On other
>>>>>> systems - say OSX - the situation is less clear to me. According to what I
>>>>>> read here
>>>>>> http://nipy.org/dipy/installation.html#openmp-with-osx
>>>>>> the OSX installation steps are not meant for standard end users. Are
>>>>>> those instructions updated?
>>>>>> As a test of that, we've just tried to skip the steps described above
>>>>>> and instead to install gcc with conda on OSX ("conda install gcc"). In the
>>>>>> process, conda installed the recent gcc-4.8 with libgomp, which seems good
>>>>>> news. Unfortunately, when we tried to compile a simple example of Cython
>>>>>> code using parallelization (see below), the process failed (fatal error:
>>>>>> limits.h : no such file or directory)...
>>>>>>
>>>>>> For the reasons above, I am wondering whether the very simple
>>>>>> solution of using the "multiprocessing" module, available from the standard
>>>>>> Python library, may be an acceptable first step towards the more efficient
>>>>>> multithreading of Cython/libgomp. With "multiprocessing", there is no extra
>>>>>> dependency on libgomp, or recent gcc or else. Moreover, multiprocessing
>>>>>> does not require to have Cython code, because it works on plain Python too.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Emanuele
>>>>>>
>>>>>> ---- test.pyx ----
>>>>>> from cython import parallel
>>>>>> from libc.stdio cimport printf
>>>>>>
>>>>>> def test_func():
>>>>>> cdef int thread_id = -1
>>>>>> with nogil, parallel.parallel(num_threads=10):
>>>>>> thread_id = parallel.threadid()
>>>>>> printf("Thread ID: %d\n", thread_id)
>>>>>> -----
>>>>>>
>>>>>> ----- setup.py -----
>>>>>> from distutils.core import setup, Extension
>>>>>> from Cython.Build import cythonize
>>>>>>
>>>>>> extensions = [Extension(
>>>>>> "test",
>>>>>> sources=["test.pyx"],
>>>>>> extra_compile_args=["-fopenmp"],
>>>>>> extra_link_args=["-fopenmp"]
>>>>>> )]
>>>>>>
>>>>>> setup(
>>>>>> ext_modules = cythonize(extensions)
>>>>>> )
>>>>>> ----
>>>>>> python setup.py build_ext --inplace
>>>>>>
>>>>>> On Tue, Dec 13, 2016 at 11:17 PM, Eleftherios Garyfallidis <
>>>>>> elef at indiana.edu> wrote:
>>>>>>
>>>>>> Hi Emanuele,
>>>>>>
>>>>>> Here is an example of how we calculated the distance matrix in
>>>>>> parallel (for the MDF) using OpenMP
>>>>>> https://github.com/nipy/dipy/blob/master/dipy/align/bundlemin.pyx
>>>>>>
>>>>>> You can just add another function that does the same using mam. It
>>>>>> should be really easy to implement as we have
>>>>>> already done it for the MDF for speeding up SLR.
>>>>>>
>>>>>> Then we need to update the bundle_distances* functions to use the
>>>>>> parallel versions.
>>>>>>
>>>>>> I'll be happy to help you with this. Let's try to schedule some time
>>>>>> to look at this together.
>>>>>>
>>>>>> Best regards,
>>>>>> Eleftherios
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 12, 2016 at 11:16 AM Emanuele Olivetti <olivetti at fbk.eu>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I usually compute the distance matrix between two lists of
>>>>>> streamlines using bundle_distances_mam() or bundle_distances_mdf(). When
>>>>>> the lists are large, it is convenient and easy to exploit the multiple
>>>>>> cores of the CPU because such computation is intrinsically (embarassingly)
>>>>>> parallel. At the moment I'm doing it through the multiprocessing or the
>>>>>> joblib modules, because I cannot find a way directly from DiPy, at least
>>>>>> according to what I see in dipy/tracking/distances.pyx . But consider that
>>>>>> I am not proficient in cython.parallel.
>>>>>>
>>>>>> Is there a preferable way to perform such parallel computation? I
>>>>>> plan to prepare a pull request in future and I'd like to be on the right
>>>>>> track.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Emanuele
>>>>>>
>>>>>> _______________________________________________
>>>>>> Neuroimaging mailing list
>>>>>> Neuroimaging at python.org
>>>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Neuroimaging mailing list
>>>>>> Neuroimaging at python.org
>>>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Neuroimaging mailing list
>>>>>> Neuroimaging at python.org
>>>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Neuroimaging mailing list
>>>>> Neuroimaging at python.org
>>>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Neuroimaging mailing list
>>> Neuroimaging at python.org
>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>
>>>
>>
>> _______________________________________________
>> Neuroimaging mailing list
>> Neuroimaging at python.org
>> https://mail.python.org/mailman/listinfo/neuroimaging
>>
>>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20161215/2d67f0b3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.pyx
Type: text/x-python
Size: 241 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20161215/2d67f0b3/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 326 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20161215/2d67f0b3/attachment-0001.py>
More information about the Neuroimaging
mailing list