[IPython-dev] [parallel] @lview.parallel() and .map
Francesco Montesano
franz.bergesund at gmail.com
Fri Sep 20 09:50:56 EDT 2013
Dear all,
I've a question about running a function decorated with @lview.parallel().
[python 3.3, ipython 1.1.0]
My function look like
def do_operation(fname):
> "read file and do sum operation"
> col = np.loadtxt(fname, usecols=[1,])
> return col.size, col.sum()
@lview.parallel(block=True)
> def func(fnames)
> "do some operation over the file names"
> oplist = []
> for fn in fnames:
> oplist.append(do_operation(fn))
> return oplist
According to the
documentation<http://ipython.org/ipython-doc/dev/parallel/parallel_multiengine.html#remote-function-decorators>
I
can call the above function as
1. res = func(fnlist)
2. res = func.map(fnlist)
where fnlist is a list of file names (so strings)
fnlist = ['/abspath/file1', '/abspath/file2', '/abspath/file3',
'/abspath/file4']
Way 1. works and I get something like
> res = [ [15,45],
> [14,42],
> [12,44],
> [17,46]]
that then I convert to numpy array
But from the example in the above link, it seems to me that can happen to
get some extra grouping (see the example "echo(range(5))")
Way 2 seems safer when it comes to execution order and result recollection,
BUT when I run it I get a bunch of these errors:
[5:apply]:
> ---------------------------------------------------------------------------
> IsADirectoryError Traceback (most recent call
> last)<string> in <module>()
> /home/.local/lib/python3.3/site-packages/ipython-1.1.0-py3.3.egg/IPython/parallel/client/remotefunction.py
> in <lambda>(f, *sequences)
> 239 if self._mapping:
> 240 if sys.version_info[0] >= 3:
> --> 241 f = lambda f, *sequences: list(map(f,
> *sequences))
> 242 else:
> 243 f = map
> <ipython-input-61-740416948047> in get_sums(flist)
> <ipython-input-1-23acb8d29a27> in do_sums(fname)
> [...]
> ----> 6 col = np.loadtxt(fname, usecols=[1,])
> [...]
> /usr/lib/python3/dist-packages/numpy/lib/npyio.py in loadtxt(fname, dtype,
> comments, delimiter, converters, skiprows, usecols, unpack, ndmin)
> 713 fh = iter(bz2.BZ2File(fname))
> 714 else:
> --> 715 fh = iter(open(fname, 'U'))
> 716 else:
> 717 fh = iter(fname)
> IsADirectoryError: [Errno 21] Is a directory: '/'
So it seems that "map" breaks up the single strings in the list of file
names.
I know that strings are iterables, but here "map" is breaking up first the
list, and then the strings. I don't know if this is the intended behaviour,
but for me this is a bug.
thanks for the support and THE ipython
Cheers,
Fra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130920/f2512702/attachment.html>
More information about the IPython-dev
mailing list