pip installation for pypy is slow
Dear pypy-devs, The pip installation of pypy (pypy -m pip install scipy, for example) is slower than python-pip clean. Is there a way to fix this problem ? Bravo! Hao Wang
On 29/11/21 6:01 am, Hao Wang wrote:
Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy. A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/
PyPy could contain some workaround to still work if you are lucky, but it is not done so far; better fix the bug in
Thanks Matti, I switched to Cython due to PyPy's incompatibility with pandas (The error is given below. If you know how to fix this, please let me know.) cpyext, the emulation layer, detected that while it is calling an object's tp_dealloc, the C code calls back a function that tries to recreate the PyPy version of the object. Usually it means that tp_dealloc calls some general PyXxx() API. It is a dangerous and potentially buggy thing to do: even in CPython the PyXxx() function could, in theory, cause a reference to the object to be taken and stored somewhere, for an amount of time exceeding tp_dealloc itself. Afterwards, the object will be freed, making that reference point to garbage. the CPython extension.
This object is of type 'pandas._libs.parsers.TextReader' Aborted
From: pypy-dev <pypy-dev-bounces+haow85=live.com@python.org> on behalf of Matti Picus <matti.picus@gmail.com> Sent: Monday, November 29, 2021 12:31 AM To: pypy-dev@python.org <pypy-dev@python.org> Subject: Re: [pypy-dev] pip installation for pypy is slow On 29/11/21 6:01 am, Hao Wang wrote:
Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy. A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/ _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 29/11/21 6:01 am, Hao Wang wrote:
Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy. A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/
PyPy could contain some workaround to still work if you are lucky, but it is not done so far; better fix the bug in
Thanks Matti, I switched to Cython due to PyPy's incompatibility with pandas (The error is given below. If you know how to fix this, please let me know.) cpyext, the emulation layer, detected that while it is calling an object's tp_dealloc, the C code calls back a function that tries to recreate the PyPy version of the object. Usually it means that tp_dealloc calls some general PyXxx() API. It is a dangerous and potentially buggy thing to do: even in CPython the PyXxx() function could, in theory, cause a reference to the object to be taken and stored somewhere, for an amount of time exceeding tp_dealloc itself. Afterwards, the object will be freed, making that reference point to garbage. the CPython extension.
This object is of type 'pandas._libs.parsers.TextReader' Aborted
From: pypy-dev <pypy-dev-bounces+haow85=live.com@python.org> on behalf of Matti Picus <matti.picus@gmail.com> Sent: Monday, November 29, 2021 12:31 AM To: pypy-dev@python.org <pypy-dev@python.org> Subject: Re: [pypy-dev] pip installation for pypy is slow On 29/11/21 6:01 am, Hao Wang wrote:
Thanks for giving PyPy a try. When pre-compiled binary wheels are not available, python must compile from source. This takes time. We have been making an effort to convince project maintainers to provide binary wheels for PyPy. SciPy has a complicated workflow and a skeleton support team. That together with the fact that currently PyPy + SciPy is much slower to run CI (small snippets of code run in tests are not PyPy's strong point, and the c-api emulation layer makes things worse) mean that SciPy is hesitant to add support for PyPy, see the PR where the CI runs were removed [0] in 2019. Our long-term proposal to solve this problem is to get pybind11 (for c++ code), cython (for c code), and f2py (for fortran code) to generate wrappers using HPy [1] instead of the C-API, which will speed up testing with PyPy. A second possibility is that the scientific python community will move away from its heavy dependency on the CPython C-API and more towards using higher-level protocols [4]. Both of these efforts will take many years unless heavily sponsored. A more specific short-term solution for your installation problem is to use conda-forge's miniforge [3]. This will solve the installation problem, but PyPy will still be slow on scipy-heavy workflows. The conda-forge project supplies binary packages for pypy3.7, and is discussing pypy3.8 support [2]: conda create -n pypy3.7 pypy conda activate pypy3.7 conda install scipy Matti [0] https://github.com/scipy/scipy/pull/10085 [1] https://hpyproject.org/ [2] https://github.com/conda-forge/conda-forge-pinning-feedstock/issues/2089 [3] https://github.com/conda-forge/miniforge [4] https://labs.quansight.org/blog/2021/11/pydata-extensibility-vision/ _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
participants (2)
-
Hao Wang -
Matti Picus