ot.emd limitations on matrix size
Hi, I'm trying to use the ot.emd function to solve for a 3000 x 5000 transportation problem, but I get all zeros for the solution. It works if I take a subset of, say, 1000 x 1000. Do anybody know what are the matrix size limitations for this function? Is it expected to handle matrices of these sizes? Thanks, Vanessa
Hello Vanessa and thank you for your interest in POT. emd on very large matrices is a very interesting problem. To be frank we did not try it over 1000x1000 The emd solver is based on C code and to my shame there are some parameters that are still hard coded here https://github.com/rflamary/POT/blob/master/ot/lp/EMD_wrapper.cpp such that the max number of iteration line 22 there are probably other parameters in the code that would help the problem work in very large matrices. You can have a look and change them before building POT again (be carefull to uninstall is with pip before). Feel free to open an Issue on github with the code that reproduce the bug, since it's summer I don't know if anybody will be available on short notice. I know Antoine rolet has been working on an improvement to the emd wrapper here https://github.com/arolet/POT Rémi Le 09/08/2017 à 18:56, Vanessa Ortiz a écrit :
Hi,
I'm trying to use the ot.emd function to solve for a 3000 x 5000 transportation problem, but I get all zeros for the solution. It works if I take a subset of, say, 1000 x 1000. Do anybody know what are the matrix size limitations for this function? Is it expected to handle matrices of these sizes?
Thanks,
Vanessa
_______________________________________________ POT mailing list pot@python.org https://mail.python.org/mm3/mailman3/lists/pot.python.org/
-- Rémi Flamary Web: http://remi.flamary.com Tel: +33 (0)4 92 07 63 80 Laboratoire Lagrange, UMR CNRS 7293 Observatoire de la Côte d'Azur Université de Nice Sophia-Antipolis
Hi Vanessa, A little side-comment on rémi’s reply after my holiday break: what is the POT version that you are using ? We experimented some issues with the EMD solver on Windows 64 architectures that were solved in release 0.3.1. Are you using this last release ? Best regards, Nicolas
Le 10 août 2017 à 08:21, Rémi Flamary <remi.flamary@unice.fr> a écrit :
Hello Vanessa and thank you for your interest in POT.
emd on very large matrices is a very interesting problem. To be frank we did not try it over 1000x1000
The emd solver is based on C code and to my shame there are some parameters that are still hard coded here https://github.com/rflamary/POT/blob/master/ot/lp/EMD_wrapper.cpp such that the max number of iteration line 22
there are probably other parameters in the code that would help the problem work in very large matrices. You can have a look and change them before building POT again (be carefull to uninstall is with pip before).
Feel free to open an Issue on github with the code that reproduce the bug, since it's summer I don't know if anybody will be available on short notice. I know Antoine rolet has been working on an improvement to the emd wrapper here https://github.com/arolet/POT
Rémi
Le 09/08/2017 à 18:56, Vanessa Ortiz a écrit :
Hi, I'm trying to use the ot.emd function to solve for a 3000 x 5000 transportation problem, but I get all zeros for the solution. It works if I take a subset of, say, 1000 x 1000. Do anybody know what are the matrix size limitations for this function? Is it expected to handle matrices of these sizes? Thanks, Vanessa _______________________________________________ POT mailing list pot@python.org https://mail.python.org/mm3/mailman3/lists/pot.python.org/
-- Rémi Flamary Web: http://remi.flamary.com Tel: +33 (0)4 92 07 63 80 Laboratoire Lagrange, UMR CNRS 7293 Observatoire de la Côte d'Azur Université de Nice Sophia-Antipolis
_______________________________________________ POT mailing list pot@python.org https://mail.python.org/mm3/mailman3/lists/pot.python.org/
Hello Vanessa and thank you for your interest in POT. emd on very large matrices is a very interesting problem. To be frank we did not try it over 1000x1000 The emd solver is based on C code and to my shame there are some parameters that are still hard coded here https://github.com/rflamary/POT/blob/master/ot/lp/EMD_wrapper.cpp such that the max number of iteration line 22 there are probably other parameters in the code that would help the problem work in very large matrices. You can have a look and change them before building POT again (be carefull to uninstall is with pip before). Feel free to open an Issue on github with the code that reproduce the bug, since it's summer I don't know if anybody will be available on short notice. I know Antoine rolet has been working on an improvement to the emd wrapper here https://github.com/arolet/POT Rémi
participants (4)
-
Nicolas Courty -
remi.flamary＠unice.fr -
Rémi Flamary
-
Vanessa Ortiz