<div dir="ltr">Hi Eleftherios,<div><br></div><div>Thank you for pointing me to the MDF example. From what I see the Cython syntax is not complex, which is good.</div><div><br></div><div>My only concern is the availability of OpenMP in the systems where DiPy is used. On a reasonably recent GNU/Linux machine it seems straightforward to have libgomp and the proper version of gcc. On other systems - say OSX - the situation is less clear to me. According to what I read here</div><div> <a href="http://nipy.org/dipy/installation.html#openmp-with-osx">http://nipy.org/dipy/installation.html#openmp-with-osx</a></div><div>the OSX installation steps are not meant for standard end users. Are those instructions updated?</div><div>As a test of that, we've just tried to skip the steps described above and instead to install gcc with conda on OSX ("conda install gcc"). In the process, conda installed the recent gcc-4.8 with libgomp, which seems good news. Unfortunately, when we tried to compile a simple example of Cython code using parallelization (see below), the process failed (fatal error: limits.h : no such file or directory)...</div><div><br></div><div>For the reasons above, I am wondering whether the very simple solution of using the "multiprocessing" module, available from the standard Python library, may be an acceptable first step towards the more efficient multithreading of Cython/libgomp. With "multiprocessing", there is no extra dependency on libgomp, or recent gcc or else. Moreover, multiprocessing does not require to have Cython code, because it works on plain Python too.</div><div><br></div><div>Best,</div><div><br></div><div>Emanuele</div><div><br></div><div>---- test.pyx ----</div><div><div>from cython import parallel</div><div>from libc.stdio cimport printf</div><div><br></div><div>def test_func():</div><div> cdef int thread_id = -1</div><div> with nogil, parallel.parallel(num_threads=10):</div><div> thread_id = parallel.threadid()</div><div> printf("Thread ID: %d\n", thread_id)</div></div><div>-----</div><div><br></div><div>----- setup.py -----</div><div><div>from distutils.core import setup, Extension</div><div>from Cython.Build import cythonize</div><div><br></div><div>extensions = [Extension(</div><div> "test",</div><div> sources=["test.pyx"],</div><div> extra_compile_args=["-fopenmp"],</div><div> extra_link_args=["-fopenmp"]</div><div> )]</div><div><br></div><div>setup(</div><div> ext_modules = cythonize(extensions)</div><div>)</div></div><div>----</div><div>python setup.py build_ext --inplace<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 13, 2016 at 11:17 PM, Eleftherios Garyfallidis <span dir="ltr"><<a href="mailto:elef@indiana.edu" target="_blank">elef@indiana.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Emanuele,<div><br></div><div>Here is an example of how we calculated the distance matrix in parallel (for the MDF) using OpenMP</div><div><a href="https://github.com/nipy/dipy/blob/master/dipy/align/bundlemin.pyx" target="_blank">https://github.com/nipy/dipy/<wbr>blob/master/dipy/align/<wbr>bundlemin.pyx</a><br></div><div><br></div><div>You can just add another function that does the same using mam. It should be really easy to implement as we have</div><div>already done it for the MDF for speeding up SLR.</div><div><br></div><div>Then we need to update the bundle_distances* functions to use the parallel versions.</div><div><br></div><div>I'll be happy to help you with this. Let's try to schedule some time to look at this together.</div><div><br></div><div>Best regards,</div><div>Eleftherios</div><div><br></div></div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Mon, Dec 12, 2016 at 11:16 AM Emanuele Olivetti <<a href="mailto:olivetti@fbk.eu" target="_blank">olivetti@fbk.eu</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr" class="m_3968506265955238318gmail_msg">Hi,<div class="m_3968506265955238318gmail_msg"><br class="m_3968506265955238318gmail_msg"></div><div class="m_3968506265955238318gmail_msg">I usually compute the distance matrix between two lists of streamlines using bundle_distances_mam() or bundle_distances_mdf(). When the lists are large, it is convenient and easy to exploit the multiple cores of the CPU because such computation is intrinsically (embarassingly) parallel. At the moment I'm doing it through the multiprocessing or the joblib modules, because I cannot find a way directly from DiPy, at least according to what I see in dipy/tracking/distances.pyx . But consider that I am not proficient in cython.parallel.</div><div class="m_3968506265955238318gmail_msg"><br class="m_3968506265955238318gmail_msg"></div><div class="m_3968506265955238318gmail_msg">Is there a preferable way to perform such parallel computation? I plan to prepare a pull request in future and I'd like to be on the right track.</div><div class="m_3968506265955238318gmail_msg"><br class="m_3968506265955238318gmail_msg"></div><div class="m_3968506265955238318gmail_msg">Best,</div><div class="m_3968506265955238318gmail_msg"><br class="m_3968506265955238318gmail_msg"></div><div class="m_3968506265955238318gmail_msg">Emanuele</div><div class="m_3968506265955238318gmail_msg"><br class="m_3968506265955238318gmail_msg"></div></div></div></div>
______________________________<wbr>_________________<br class="m_3968506265955238318gmail_msg">
Neuroimaging mailing list<br class="m_3968506265955238318gmail_msg">
<a href="mailto:Neuroimaging@python.org" class="m_3968506265955238318gmail_msg" target="_blank">Neuroimaging@python.org</a><br class="m_3968506265955238318gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" class="m_3968506265955238318gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/neuroimaging</a><br class="m_3968506265955238318gmail_msg">
</blockquote></div>
<br>______________________________<wbr>_________________<br>
Neuroimaging mailing list<br>
<a href="mailto:Neuroimaging@python.org">Neuroimaging@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/neuroimaging" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/neuroimaging</a><br>
<br></blockquote></div><br></div>