<div dir="ltr">Dear all,<div><br></div><div>I've a question about running a function decorated with @lview.parallel(). [python 3.3, ipython 1.1.0]</div><div><br></div><div>My function look like</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
def do_operation(fname):<br> "read file and do sum operation"<br> col = np.loadtxt(fname, usecols=[1,])<br> return col.size, col.sum()</blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
@lview.parallel(block=True)<br>def func(fnames)<br> "do some operation over the file names"<br> oplist = []<br> for fn in fnames:<br> oplist.append(do_operation(fn))<br> return oplist</blockquote>
<div><br></div><div>According to <a href="http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#remote-function-decorators">the documentation</a> I can call the above function as </div><div>1. res = func(fnlist)</div>
<div>2. res = func.map(fnlist)</div><div>where fnlist is a list of file names (so strings) </div><div>fnlist = ['/abspath/file1', '/abspath/file2', '/abspath/file3', '/abspath/file4']</div>
<div><br></div><div>Way 1. works and I get something like </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
res = [ [15,45],<br> [14,42], <br> [12,44], <br> [17,46]]</blockquote><div>that then I convert to numpy array</div><div><br></div><div>But from the example in the above link, it seems to me that can happen to get some extra grouping (see the example "echo(range(5))")</div>
<div><br></div><div>Way 2 seems safer when it comes to execution order and result recollection, BUT when I run it I get a bunch of these errors:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
[5:apply]: <br>---------------------------------------------------------------------------<br>IsADirectoryError Traceback (most recent call last)<string> in <module>()<br>/home/.local/lib/python3.3/site-packages/ipython-1.1.0-py3.3.egg/IPython/parallel/client/remotefunction.py in <lambda>(f, *sequences)<br>
239 if self._mapping:<br> 240 if sys.version_info[0] >= 3:<br>--> 241 f = lambda f, *sequences: list(map(f, *sequences))<br> 242 else:<br> 243 f = map<br>
<ipython-input-61-740416948047> in get_sums(flist)<br><ipython-input-1-23acb8d29a27> in do_sums(fname)<br>[...]<br>----> 6 col = np.loadtxt(fname, usecols=[1,])<br>[...]<br>/usr/lib/python3/dist-packages/numpy/lib/npyio.py in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin)<br>
713 fh = iter(bz2.BZ2File(fname))<br> 714 else:<br>--> 715 fh = iter(open(fname, 'U'))<br> 716 else:<br> 717 fh = iter(fname)<br>IsADirectoryError: [Errno 21] Is a directory: '/'</blockquote>
</div><div><br></div><div>So it seems that "map" breaks up the single strings in the list of file names. </div><div><br></div><div><br></div><div>I know that strings are iterables, but here "map" is breaking up first the list, and then the strings. I don't know if this is the intended behaviour, but for me this is a bug. </div>
<div><br></div><div>thanks for the support and THE ipython</div><div><br></div><div>Cheers,</div><div><br></div><div>Fra</div><div><br></div></div>