<div dir="ltr">Hi,<div><br></div><div>The instructions at <a href="http://nipy.org/dipy/installation.html#openmp-with-osx" class="gmail-m_-6038801414603531625gmail_msg" target="_blank" style="font-size:12.8px">http://nipy.org/dipy/<wbr>installation.html#openmp-with-<wbr>osx</a> are outdated since the clang-omp formula does not exist anymore.</div><div><br></div>With the release of Clang 3.8.0 (08 Mar 2016), OpenMP 3.1 support is enabled in Clang by default. You will need the -fopenmp=libomp flag while building.<div><br><div>I started a while back on a DIPY homebrew formula to allow installation via "brew install dipy", see <a href="https://github.com/Homebrew/homebrew-python/pull/310">https://github.com/Homebrew/homebrew-python/pull/310</a>. However it was based around the deprecated clang-omp and I didn't get around to fix it. Might look into it soon if I find some spare time to tweak the formula.</div></div><div><br></div><div>Regards,</div><div>Stephan</div><div class="gmail_extra"><br><div class="gmail_quote">2016-12-14 17:14 GMT+01:00 Samuel St-Jean <span dir="ltr"><<a href="mailto:stjeansam@gmail.com" target="_blank">stjeansam@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>That also depends on which version of clang ships by default with osx, in which case you have to play around with it to get a new one. I think it starts with clang 3.7 to have openmp support (I only ever tried Mac osx 10.9, so anyone more experienced can chime in), but everything older has to go through the hombebrew gcc install and company. Might be worhtwhile to check if openmp support is out of the box now, and starting on which mac osx versions, since older ones could be problematic for first time installs. <br><br>I also have to admit I have no idea how old is old in the mac world, so maybe 10.9 is already phased out by now, but it was a hard and time consuming building around stuff with homebrew experience (and again, first time I used a mac, so, I guess the average user would also have some issues).<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888">Samuel<br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-12-14 16:51 GMT+01:00 Eleftherios Garyfallidis <span dir="ltr"><<a href="mailto:elef@indiana.edu" target="_blank">elef@indiana.edu</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Hi Emanuele,</div><div><br></div>My understanding is that openmp was only temporarily not available when clang replaced gcc in osx. <div><br></div><div>So, I would suggest to go ahead with openmp. Any current installation issues are only temporarily for osx. </div><div>Openmp gives us a lot of capability to play with shared memory and it is a standard that will be around</div><div>for very long time. Also, the great integration in cython makes the algorithms really easy to read. </div><div>So, especially for this project my recommendation is to use openmp rather than multiprocessing. All the way! :)</div><div><br></div><div>I am CC'ing Stephan who wrote the instructions for osx. I am sure he can help you with this. I would also suggest</div><div>to check if xcode provides any new guis for enabling openmp. I remember there was something for that.</div><div><br></div><div>Laterz!</div><span class="m_-4312361255011100872HOEnZb"><font color="#888888"><div>Eleftherios</div></font></span><div><div class="m_-4312361255011100872h5"><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Dec 14, 2016 at 6:29 AM Emanuele Olivetti <<a href="mailto:olivetti@fbk.eu" target="_blank">olivetti@fbk.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Hi Eleftherios,<div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Thank you for pointing me to the MDF example. From what I see the Cython syntax is not complex, which is good.</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">My only concern is the availability of OpenMP in the systems where DiPy is used. On a reasonably recent GNU/Linux machine it seems straightforward to have libgomp and the proper version of gcc. On other systems - say OSX - the situation is less clear to me. According to what I read here</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> <a href="http://nipy.org/dipy/installation.html#openmp-with-osx" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">http://nipy.org/dipy/install<wbr>ation.html#openmp-with-osx</a></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">the OSX installation steps are not meant for standard end users. Are those instructions updated?</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">As a test of that, we've just tried to skip the steps described above and instead to install gcc with conda on OSX ("conda install gcc"). In the process, conda installed the recent gcc-4.8 with libgomp, which seems good news. Unfortunately, when we tried to compile a simple example of Cython code using parallelization (see below), the process failed (fatal error: limits.h : no such file or directory)...</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">For the reasons above, I am wondering whether the very simple solution of using the "multiprocessing" module, available from the standard Python library, may be an acceptable first step towards the more efficient multithreading of Cython/libgomp. With "multiprocessing", there is no extra dependency on libgomp, or recent gcc or else. Moreover, multiprocessing does not require to have Cython code, because it works on plain Python too.</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Best,</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Emanuele</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">---- test.pyx ----</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">from cython import parallel</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">from libc.stdio cimport printf</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">def test_func():</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> cdef int thread_id = -1</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> with nogil, parallel.parallel(num_threads=<wbr>10):</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> thread_id = parallel.threadid()</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> printf("Thread ID: %d\n", thread_id)</div></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">-----</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">----- setup.py -----</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">from distutils.core import setup, Extension</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">from Cython.Build import cythonize</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">extensions = [Extension(</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> "test",</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> sources=["test.pyx"],</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> extra_compile_args=["-fopenmp"<wbr>],</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> extra_link_args=["-fopenmp"]</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> )]</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">setup(</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"> ext_modules = cythonize(extensions)</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">)</div></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">----</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">python setup.py build_ext --inplace<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div></div><div class="gmail_extra m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="gmail_quote m_-4312361255011100872m_-4197421230930321653gmail_msg">On Tue, Dec 13, 2016 at 11:17 PM, Eleftherios Garyfallidis <span dir="ltr" class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><<a href="mailto:elef@indiana.edu" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">elef@indiana.edu</a>></span> wrote:<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><blockquote class="gmail_quote m_-4312361255011100872m_-4197421230930321653gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Hi Emanuele,<div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Here is an example of how we calculated the distance matrix in parallel (for the MDF) using OpenMP</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><a href="https://github.com/nipy/dipy/blob/master/dipy/align/bundlemin.pyx" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">https://github.com/nipy/dipy/b<wbr>lob/master/dipy/align/bundlemi<wbr>n.pyx</a><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">You can just add another function that does the same using mam. It should be really easy to implement as we have</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">already done it for the MDF for speeding up SLR.</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Then we need to update the bundle_distances* functions to use the parallel versions.</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">I'll be happy to help you with this. Let's try to schedule some time to look at this together.</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Best regards,</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg">Eleftherios</div><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div></div><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="gmail_quote m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917h5 m_-4312361255011100872m_-4197421230930321653gmail_msg"><div dir="ltr" class="m_-4312361255011100872m_-4197421230930321653gmail_msg">On Mon, Dec 12, 2016 at 11:16 AM Emanuele Olivetti <<a href="mailto:olivetti@fbk.eu" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">olivetti@fbk.eu</a>> wrote:<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div></div></div><blockquote class="gmail_quote m_-4312361255011100872m_-4197421230930321653gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-4312361255011100872m_-4197421230930321653gmail_msg"><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917h5 m_-4312361255011100872m_-4197421230930321653gmail_msg"><div dir="ltr" class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">Hi,<div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">I usually compute the distance matrix between two lists of streamlines using bundle_distances_mam() or bundle_distances_mdf(). When the lists are large, it is convenient and easy to exploit the multiple cores of the CPU because such computation is intrinsically (embarassingly) parallel. At the moment I'm doing it through the multiprocessing or the joblib modules, because I cannot find a way directly from DiPy, at least according to what I see in dipy/tracking/distances.pyx . But consider that I am not proficient in cython.parallel.</div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">Is there a preferable way to perform such parallel computation? I plan to prepare a pull request in future and I'd like to be on the right track.</div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">Best,</div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"></div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">Emanuele</div><div class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg"></div></div></div></div>
______________________________<wbr>_________________<br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">
Neuroimaging mailing list<br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="mailto:Neuroimaging@python.org" class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">Neuroimaging@python.org</a><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/neuroimaging</a><br class="m_-4312361255011100872m_-4197421230930321653m_-3294861539979494917m_3968506265955238318gmail_msg m_-4312361255011100872m_-4197421230930321653gmail_msg">
</blockquote></div>
<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">______________________________<wbr>_________________<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
Neuroimaging mailing list<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="mailto:Neuroimaging@python.org" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">Neuroimaging@python.org</a><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/neuroimaging</a><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></blockquote></div><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg"></div>
______________________________<wbr>_________________<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
Neuroimaging mailing list<br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="mailto:Neuroimaging@python.org" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">Neuroimaging@python.org</a><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" class="m_-4312361255011100872m_-4197421230930321653gmail_msg" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/neuroimaging</a><br class="m_-4312361255011100872m_-4197421230930321653gmail_msg">
</blockquote></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Neuroimaging mailing list<br>
<a href="mailto:Neuroimaging@python.org" target="_blank">Neuroimaging@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/neuroimaging</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>