Multiprocessing - CPython and PyPy
Hi everyone. Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then the subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy , but there it says that the procedure only works with CPython 3.4. I wonder is there is any clean direct way to do this. Appreciate any help. -- Luis J. Novoa
Hi Luis. Multiprocessing works under pypy, so just run your program under pypy with no changes and see what happens On Tue, Nov 24, 2015 at 7:30 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi everyone.
Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then the subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy , but there it says that the procedure only works with CPython 3.4.
I wonder is there is any clean direct way to do this.
Appreciate any help.
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Maciej, Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy. On Tue, Nov 24, 2015 at 3:23 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Luis.
Multiprocessing works under pypy, so just run your program under pypy with no changes and see what happens
On Tue, Nov 24, 2015 at 7:30 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi everyone.
Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then the subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy , but there it says that the procedure only works with CPython 3.4.
I wonder is there is any clean direct way to do this.
Appreciate any help.
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
-- Luis J. Novoa
you can try using execnet? On Tue, Nov 24, 2015 at 10:35 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
On Tue, Nov 24, 2015 at 3:23 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Luis.
Multiprocessing works under pypy, so just run your program under pypy with no changes and see what happens
On Tue, Nov 24, 2015 at 7:30 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi everyone.
Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then the subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy , but there it says that the procedure only works with CPython 3.4.
I wonder is there is any clean direct way to do this.
Appreciate any help.
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
-- Luis J. Novoa
In a message of Tue, 24 Nov 2015 15:35:26 -0500, Luis José Novoa writes:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
You've tried and it doesn't work? Gorobipy wants numpy in order to work, but the question is _how much numpy_? If you are lucky, the parts it needs are already in pypy's numpy support. Laura
I get the following: \Downloads\pypy-4.0.0-win32\pypy-4.0.0-win32\site-packages\gurobipy\__init__.py", line 1, in <module> from .gurobipy import * ImportError: No module named gurobipy.gurobipy After installing it as pypy setup,py install On Tue, Nov 24, 2015 at 3:45 PM, Laura Creighton <lac@openend.se> wrote:
In a message of Tue, 24 Nov 2015 15:35:26 -0500, Luis José Novoa writes:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
You've tried and it doesn't work? Gorobipy wants numpy in order to work, but the question is _how much numpy_? If you are lucky, the parts it needs are already in pypy's numpy support.
Laura
-- Luis J. Novoa
Maybe Im doing something wrong? Appreciate all the help [image: Inline image 1] On Tue, Nov 24, 2015 at 3:45 PM, Laura Creighton <lac@openend.se> wrote:
In a message of Tue, 24 Nov 2015 15:35:26 -0500, Luis José Novoa writes:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
You've tried and it doesn't work? Gorobipy wants numpy in order to work, but the question is _how much numpy_? If you are lucky, the parts it needs are already in pypy's numpy support.
Laura
-- Luis J. Novoa
your install looks somehow wrong. I would suggest you use virtualenv On Wed, Nov 25, 2015 at 12:38 AM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Maybe Im doing something wrong? Appreciate all the help
[image: Inline image 1]
On Tue, Nov 24, 2015 at 3:45 PM, Laura Creighton <lac@openend.se> wrote:
In a message of Tue, 24 Nov 2015 15:35:26 -0500, Luis José Novoa writes:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
You've tried and it doesn't work? Gorobipy wants numpy in order to work, but the question is _how much numpy_? If you are lucky, the parts it needs are already in pypy's numpy support.
Laura
-- Luis J. Novoa
fwiw, you might want to look at this: https://github.com/felipecruz/gurobi_cffi from what I read in the readme, it probably exposes a different interface than gurobipy, but it might enough for your use case. On Tue, Nov 24, 2015 at 9:35 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
On Tue, Nov 24, 2015 at 3:23 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Luis.
Multiprocessing works under pypy, so just run your program under pypy with no changes and see what happens
Hi everyone.
Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then
On Tue, Nov 24, 2015 at 7:30 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote: the
subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy , but there it says that the procedure only works with CPython 3.4.
I wonder is there is any clean direct way to do this.
Appreciate any help.
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Thank you for your reply. I've looked into https://github.com/felipecruz/gurobi_cffi before, and, unfortunately, it does not have the full capabilities of gurobipy LJN On Nov 25, 2015 8:05 AM, "Antonio Cuni" <anto.cuni@gmail.com> wrote:
fwiw, you might want to look at this: https://github.com/felipecruz/gurobi_cffi
from what I read in the readme, it probably exposes a different interface than gurobipy, but it might enough for your use case.
On Tue, Nov 24, 2015 at 9:35 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote:
Hi Maciej,
Thanks for your reply. The problem is that Im using gurobipy (for the interaction with the mathematical programming solver Gurobi) to solve a master problem and then using the output to solve the subproblems. Now, gurobipy is not compatible wiith PyPy.
On Tue, Nov 24, 2015 at 3:23 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Luis.
Multiprocessing works under pypy, so just run your program under pypy with no changes and see what happens
Hi everyone.
Im trying to solve some problems in parallel using the multiprocessing module in Python 2.7. A general model is built using CPython and then
On Tue, Nov 24, 2015 at 7:30 PM, Luis José Novoa <luisjosenovoa@gmail.com> wrote: the
subproblems are solved in parallel returning results in a queue. This is currently working fine. I would like to solve the subproblems using PyPy to increase speed. I found
http://project-trains.tumblr.com/post/102076598295/multiprocessing-pypy ,
but there it says that the procedure only works with CPython 3.4.
I wonder is there is any clean direct way to do this.
Appreciate any help.
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
-- Luis J. Novoa
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
participants (4)
-
Antonio Cuni -
Laura Creighton -
Luis José Novoa -
Maciej Fijalkowski