[IPython-dev] Using IPython Parallel inside a class
Michael McKerns
mmckerns at caltech.edu
Wed Dec 3 20:58:02 EST 2014
Try replacing the serialization engine with `dill` or `cloudpickle`.
That's probably your best chance at serializing and shipping the
objects across the wire. IPython.parallel lets you override the
serializer with either of these.
> On Wed, Dec 3, 2014 at 12:37 PM, Rafał Skolasiński
> <rjtskolasinski at gmail.com
>> wrote:
>
>> Hi,
>>
>> It doesn't look like a solution. I still get bunch of errors. And I
>> would
>> like to use self.hamiltonian instance, not create more instances of it.
>> the
>> hamiltonian object in my real (not just a test) simulation code enters
>> as
>> the argument to the Simulation class constructor.
>>
>
> Your original code gave errors trying to serialize the Hamiltonian
> instance
> in order to make copies for each machine in the cluster. Some objects
> can't
> be serialized like that. In any event, each machine needs to have access
> to
> the data that it will process. So, you need to somehow get that data (in
> whole or in part) to each machine.
>
> Hope that helps,
>
> -Doug
>
>
>
>>
>> And I would also like to execute hamiltonian.get_matrix method in
>> parallel. The version of that method I use in my real simulation code is
>> a
>> little more complex and also take some time to be executed. Not maybe as
>> much as diagonalization but still.
>>
>> Cheers,
>> Rafal
>>
>> 2014-12-03 18:28 GMT+01:00 Doug Blank <doug.blank at gmail.com>:
>>
>>> I haven't done much parallel IPython, but it looks like you just need
>>> to
>>> define/import everything on the remote clients, something like:
>>>
>>> def get_energy(momentum):
>>> import scipy.sparse.linalg as sla
>>> from scipy.sparse import coo_matrix
>>> from scipy.sparse import csc_matrix
>>> import numpy as np
>>> class Hamiltonian:
>>> def get_matrix(self, k):
>>> # sleep(3.0)
>>> return k**2 * np.diag(range(100))
>>>
>>> hamiltonian = Hamiltonian()
>>> mat = hamiltonian.get_matrix(momentum)
>>> mat = csc_matrix(mat)
>>> ev = sla.eigsh(mat,k=5, sigma = 0.5, which = 'LM')
>>> return ev[0]
>>>
>>> def compute_parallel(momenta):
>>> rc = parallel.Client()
>>> dview = rc[:]
>>> spectrum = dview.map_sync(get_energy,momenta)
>>> return spectrum
>>>
>>> -Doug
>>>
>>> On Wed, Dec 3, 2014 at 12:21 PM, Rafał Skolasiński <
>>> rjtskolasinski at gmail.com> wrote:
>>>
>>>> Oh, in the file I attached there should ofc be line
>>>> spectrum = dview.map_sync(self.get_energy,momenta)
>>>> instead of
>>>> spectrum = dview.map_sync(self.get_energy_for_parallel,momenta)
>>>>
>>>> Sorry for mistake.
>>>> Raphael
>>>>
>>>>
>>>> 2014-12-03 18:05 GMT+01:00 Rafał Skolasiński
>>>> <rjtskolasinski at gmail.com>:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> I hope this is the right place to ask this kind of question.
>>>>>
>>>>> I would like to use IPython parallel in my physical simulation.
>>>>> In my mind I've got class Simulation, which contain as a field object
>>>>> hamiltonian of class Hamiltonian.
>>>>>
>>>>> Method get_energy should call hamiltonian.get_matrix(momentum),
>>>>> diagonalize it with scipy method and return energies.
>>>>>
>>>>> I would like to execute it in parallel for a list of different
>>>>> momenta.
>>>>>
>>>>>
>>>>> I tested it with a standard python 'map' function and it worked so I
>>>>> thought that easiest way would be to use 'dview.map_sync' in the same
>>>>> way
>>>>> but it doesn't looks like a proper way of doing it.
>>>>>
>>>>> I am attaching my testing code. I would be very grateful for any help
>>>>> and ideas how to do it in a nice and clean way.
>>>>>
>>>>> Cheers,
>>>>> Raphael
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> IPython-dev mailing list
>>>> IPython-dev at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
---
Mike McKerns
California Institute of Technology
TEL: (626)395-5773 or (626)590-8470
http://www.its.caltech.edu/~mmckerns
mmckerns at caltech.edu
More information about the IPython-dev
mailing list