[IPython-dev] [parallel] @lview.parallel() and .map

Francesco Montesano franz.bergesund at gmail.com
Fri Sep 20 09:50:56 EDT 2013


Dear all,

I've a question about running a function decorated with @lview.parallel().
[python 3.3, ipython 1.1.0]

My function look like

def do_operation(fname):
>     "read file and do sum operation"
>     col = np.loadtxt(fname, usecols=[1,])
>     return col.size, col.sum()


@lview.parallel(block=True)
> def func(fnames)
>      "do some operation over the file names"
>      oplist = []
>      for fn in fnames:
>           oplist.append(do_operation(fn))
>      return oplist


According to the
documentation<http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#remote-function-decorators>
I
can call the above function as
1. res = func(fnlist)
2. res = func.map(fnlist)
where fnlist is a list of file names (so strings)
fnlist = ['/abspath/file1', '/abspath/file2', '/abspath/file3',
'/abspath/file4']

Way 1. works and I get something like

> res = [ [15,45],
>            [14,42],
>            [12,44],
>            [17,46]]

that then I convert to numpy array

But from the example in the above link, it seems to me that can happen to
get some extra grouping (see the example "echo(range(5))")

Way 2 seems safer when it comes to execution order and result recollection,
BUT when I run it I get a bunch of these errors:

[5:apply]:
> ---------------------------------------------------------------------------
> IsADirectoryError                         Traceback (most recent call
> last)<string> in <module>()
> /home/.local/lib/python3.3/site-packages/ipython-1.1.0-py3.3.egg/IPython/parallel/client/remotefunction.py
> in <lambda>(f, *sequences)
>     239             if self._mapping:
>     240                 if sys.version_info[0] >= 3:
> --> 241                     f = lambda f, *sequences: list(map(f,
> *sequences))
>     242                 else:
>     243                     f = map
> <ipython-input-61-740416948047> in get_sums(flist)
> <ipython-input-1-23acb8d29a27> in do_sums(fname)
> [...]
> ----> 6     col = np.loadtxt(fname, usecols=[1,])
> [...]
> /usr/lib/python3/dist-packages/numpy/lib/npyio.py in loadtxt(fname, dtype,
> comments, delimiter, converters, skiprows, usecols, unpack, ndmin)
>     713                 fh = iter(bz2.BZ2File(fname))
>     714             else:
> --> 715                 fh = iter(open(fname, 'U'))
>     716         else:
>     717             fh = iter(fname)
> IsADirectoryError: [Errno 21] Is a directory: '/'


So it seems that "map" breaks up the single strings in the list of file
names.


I know that strings are iterables, but here "map" is breaking up first the
list, and then the strings. I don't know if this is the intended behaviour,
but for me this is a bug.

thanks for the support and THE ipython

Cheers,

Fra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130920/f2512702/attachment.html>


More information about the IPython-dev mailing list