Are you using Bento to build scipy?
With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento. The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems. If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!
On Sat, Apr 1, 2017 at 4:55 AM, CJ Carey <perimosocordiae@gmail.com> wrote:
With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento.
The project seems to have been abandoned before reaching stability,
Unfortunately, yes. At this point I think it's unlikely that David will come back to working on it. But I'd like to hear his thoughts on it.
and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems.
Unfortunately, no. Distutils/setuptools will never become a build system with a sane architecture I'm afraid. Even things like proper parallel builds are unlikely to appear. Also the way BLAS/LAPACK libraries are supported in Bento is a lot better than what numpy.distutils does.
If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!
I am using it all the time, but I suspect I'm the only one. Bento is way faster than distutils for building SciPy and produces sane build logs, hence I prefer it. But it is hard to install because it depends on a specific version of Waf (which is not on PyPI) and has some unresolved issues on Python 3.x. For me it has been worth the effort to keep the Bento build working, and given that it's not in our build matrix it doesn't really bother anyone. But if others prefer to remove Bento support, I am fine with that too. Cheers, Ralf
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is way
I seem to remember Stefan van der Walt saying the same thing :-).
So, I guess there are two of us ;) Ralf wrote:
But it is hard to install because it depends on a specific version of Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster). Stéfan
On Thu, Apr 6, 2017 at 6:27 AM, Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is way
I seem to remember Stefan van der Walt saying the same thing :-).
So, I guess there are two of us ;)
Ralf wrote:
But it is hard to install because it depends on a specific version of Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster).
You mean vendor Waf in Bento and make a Bento pip-installable PyPI package right ? David and I discussed that already 2 years ago, that'd be the way to go. Shouldn't be too hard, Waf is designed to be vendored. What's a little more work is fixing some obvious issues: it doesn't work on Python 3.x right now, there was a recurring issue with -fPIC going missing, finish the open PR on producing wheels. Ralf
On Wed, Apr 5, 2017 at 3:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Apr 6, 2017 at 6:27 AM, Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is way
I seem to remember Stefan van der Walt saying the same thing :-).
So, I guess there are two of us ;)
Ralf wrote:
But it is hard to install because it depends on a specific version of Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster).
You mean vendor Waf in Bento and make a Bento pip-installable PyPI package right ? David and I discussed that already 2 years ago, that'd be the way to go. Shouldn't be too hard, Waf is designed to be vendored.
What's a little more work is fixing some obvious issues: it doesn't work on Python 3.x right now, there was a recurring issue with -fPIC going missing, finish the open PR on producing wheels.
Okay, these fixes are probably not going to materialize. And I've finally ditched Python 2.7 for good a few months ago, and have dealt with my distutils aversion by getting better hardware. I haven't maintained the Bento build over the last few months, so it's not quite working anymore. Therefore I now propose to remove Bento support for v1.2.0 Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year. Thoughts? Ralf
Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year.
It looks like you have to have CMake installed though ( http://scikit-build.readthedocs.io/en/latest/installation.html). This is a hurdle in itself.
On Mon, Jul 16, 2018 at 1:31 AM, Andrew Nelson <andyfaff@gmail.com> wrote:
Related: scikit-build seems to have quite a bit of momentum. I've never
used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year.
It looks like you have to have CMake installed though ( http://scikit-build.readthedocs.io/en/latest/installation.html). This is a hurdle in itself.
There are binary wheels (https://pypi.org/project/cmake) and it's packaged for conda, plus good binary installers - for a development setup that's about as easy as it is going to get. Ralf
Hi! I’ll be honest. One thing that has prevented me from getting into the C/C++ meat of NumPy/SciPy is exactly what scikit-build is trying to solve. I’m all for extra dev dependencies if they make development easier. Best Regards, Hameer Abbasi Sent from Astro <https://www.helloastro.com> for iOS On 16. Jul 2018 at 01:13, Ralf Gommers <ralf.gommers@gmail.com> wrote: On Wed, Apr 5, 2017 at 3:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Apr 6, 2017 at 6:27 AM, Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is way
I seem to remember Stefan van der Walt saying the same thing :-).
So, I guess there are two of us ;)
Ralf wrote:
But it is hard to install because it depends on a specific version of Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster).
You mean vendor Waf in Bento and make a Bento pip-installable PyPI package right ? David and I discussed that already 2 years ago, that'd be the way to go. Shouldn't be too hard, Waf is designed to be vendored.
What's a little more work is fixing some obvious issues: it doesn't work on Python 3.x right now, there was a recurring issue with -fPIC going missing, finish the open PR on producing wheels.
Okay, these fixes are probably not going to materialize. And I've finally ditched Python 2.7 for good a few months ago, and have dealt with my distutils aversion by getting better hardware. I haven't maintained the Bento build over the last few months, so it's not quite working anymore. Therefore I now propose to remove Bento support for v1.2.0 Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year. Thoughts? Ralf _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Hi, ma, 2018-07-16 kello 06:49 -0700, Hameer Abbasi kirjoitti:
I’ll be honest. One thing that has prevented me from getting into the C/C++ meat of NumPy/SciPy is exactly what scikit-build is trying to solve. I’m all for extra dev dependencies if they make development easier.
Can you (and maybe Ralf) be more specific --- in what way there are problems? What advantages scikit-build (which glues CMake and setuptools together) brings? Typically, `python runtests.py` or `python setup.py build_ext -i` works out of the box. Of course, distutils forces you to do CFLAGS/etc customization via environment variables, and BLAS/LAPACK selection via site.cfg which is not so convenient. Pauli
Best Regards, Hameer Abbasi Sent from Astro <https://www.helloastro.com> for iOS
On 16. Jul 2018 at 01:13, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Apr 5, 2017 at 3:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Apr 6, 2017 at 6:27 AM, Stefan van der Walt <stefanv@berkel ey.edu> wrote:
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is
way
I seem to remember Stefan van der Walt saying the same thing :- ).
So, I guess there are two of us ;)
Ralf wrote:
But it is hard to install because it depends on a specific version of
Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster).
You mean vendor Waf in Bento and make a Bento pip-installable PyPI package right ? David and I discussed that already 2 years ago, that'd be the way to go. Shouldn't be too hard, Waf is designed to be vendored.
What's a little more work is fixing some obvious issues: it doesn't work on Python 3.x right now, there was a recurring issue with -fPIC going missing, finish the open PR on producing wheels.
Okay, these fixes are probably not going to materialize. And I've finally ditched Python 2.7 for good a few months ago, and have dealt with my distutils aversion by getting better hardware. I haven't maintained the Bento build over the last few months, so it's not quite working anymore. Therefore I now propose to remove Bento support for v1.2.0
Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year.
Thoughts?
Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Mon, Jul 16, 2018 at 10:22 AM, Pauli Virtanen <pav@iki.fi> wrote:
Hi,
ma, 2018-07-16 kello 06:49 -0700, Hameer Abbasi kirjoitti:
I’ll be honest. One thing that has prevented me from getting into the C/C++ meat of NumPy/SciPy is exactly what scikit-build is trying to solve. I’m all for extra dev dependencies if they make development easier.
Can you (and maybe Ralf) be more specific --- in what way there are problems?
The most painful problem for me is the long build time - that (plus inertia) is what kept me developing on Python 2.7 for ages just so I could use Bento, while as a user I had already moved to Python 3. Proper parallel builds and incremental rebuilds are very nice to have.
What advantages scikit-build (which glues CMake and setuptools together) brings?
If I understand correctly, scikit-build doesn't use anything from setuptools - it just copied the setup.py user interface. That's still a significant mistake imho, it should have a saner UI (declarative). But that's still minor compared to the advantages it has. Beyond better builds and IDE support, also support for cross-compiling could be valuable. Finally, on a bigger picture level - distutils is badly maintained and good patches can go unmerged, while numpy.distutils is a pain to maintain. An actively developed sane build tool will be valuable to both us and the wider ecosystem.
Typically, `python runtests.py` or `python setup.py build_ext -i` works out of the box.
Of course, distutils forces you to do CFLAGS/etc customization via environment variables, and BLAS/LAPACK selection via site.cfg which is not so convenient.
Yes these are issues as well. And of course we'd start with it as an optional second build system, and I volunteer to do most of the required maintenance. Ralf
Pauli
Best Regards, Hameer Abbasi Sent from Astro <https://www.helloastro.com> for iOS
On 16. Jul 2018 at 01:13, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Apr 5, 2017 at 3:18 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Apr 6, 2017 at 6:27 AM, Stefan van der Walt <stefanv@berkel ey.edu> wrote:
On Wed, Apr 5, 2017, at 04:05, Evgeni Burovski wrote:
I am using it all the time, but I suspect I'm the only one. Bento is
way
I seem to remember Stefan van der Walt saying the same thing :- ).
So, I guess there are two of us ;)
Ralf wrote:
But it is hard to install because it depends on a specific version of
Waf (which is not on PyPI) and has some unresolved issues on Python 3.x.
How hard would it be to vendor Waf into a pip package? If this worked out the box, it would be great to development (repeated builds execute much faster).
You mean vendor Waf in Bento and make a Bento pip-installable PyPI package right ? David and I discussed that already 2 years ago, that'd be the way to go. Shouldn't be too hard, Waf is designed to be vendored.
What's a little more work is fixing some obvious issues: it doesn't work on Python 3.x right now, there was a recurring issue with -fPIC going missing, finish the open PR on producing wheels.
Okay, these fixes are probably not going to materialize. And I've finally ditched Python 2.7 for good a few months ago, and have dealt with my distutils aversion by getting better hardware. I haven't maintained the Bento build over the last few months, so it's not quite working anymore. Therefore I now propose to remove Bento support for v1.2.0
Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year.
Thoughts?
Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Mon, Jul 16, 2018, 12:53 Ralf Gommers <ralf.gommers@gmail.com> wrote:
What advantages scikit-build (which glues CMake and setuptools together) brings?
If I understand correctly, scikit-build doesn't use anything from setuptools - it just copied the setup.py user interface. That's still a significant mistake imho, it should have a saner UI (declarative). But that's still minor compared to the advantages it has.
For a bit of background here... When you run 'pip install <whatever>', and pip can't find a wheel, then it automatically downloads the source release and then invokes setup.py, while making various undocumented assumptions about exactly how setup.py works. So alternative build systems actually *have* to provide some kind of awkward setup.py interface to keep 'pip install' working. This is a significant part of why so few people try to replace distutils. (I think.)
Beyond better builds and IDE support, also support for cross-compiling could be valuable.
Finally, on a bigger picture level - distutils is badly maintained and good patches can go unmerged, while numpy.distutils is a pain to maintain. An actively developed sane build tool will be valuable to both us and the wider ecosystem.
The way distutils plugin system works, almost any changes can break people, plus the biggest issues are with the fundamental architecture. So yeah, it's pretty much impossible to make distutils better. Instead, the general goal of the packaging maintainers at this point is to get rid of distutils. The big thing that will enable this is PEP 517, which replaces setup.py with a much smaller and properly-specified interface for new build systems to implement. The hope is that if creating alternative build systems becomes an easy and supported thing to do, then better options will emerge, and we'll get out of this situation where there's one tool everyone has to use and which isn't really good for anyone. Unfortunately, support for this hasn't quite landed in pip yet. But that's the direction that things are going. -n
On Mon, Jul 16, 2018 at 3:22 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Jul 16, 2018, 12:53 Ralf Gommers <ralf.gommers@gmail.com> wrote:
What advantages scikit-build (which glues CMake and setuptools together) brings?
If I understand correctly, scikit-build doesn't use anything from setuptools - it just copied the setup.py user interface. That's still a significant mistake imho, it should have a saner UI (declarative). But that's still minor compared to the advantages it has.
For a bit of background here...
When you run 'pip install <whatever>', and pip can't find a wheel, then it automatically downloads the source release and then invokes setup.py, while making various undocumented assumptions about exactly how setup.py works. So alternative build systems actually *have* to provide some kind of awkward setup.py interface to keep 'pip install' working.
Indeed. But this is way nicer, just a minimal shim: https://cournape.github.io/Bento/html/transition.html#adding-bento-based-set... Anyway, I don't have time to spend on this, so I'm not worrying about it until after we have PEP 517. This is a significant part of why so few people try to replace distutils.
(I think.)
Beyond better builds and IDE support, also support for cross-compiling could be valuable.
Finally, on a bigger picture level - distutils is badly maintained and good patches can go unmerged, while numpy.distutils is a pain to maintain. An actively developed sane build tool will be valuable to both us and the wider ecosystem.
The way distutils plugin system works, almost any changes can break people, plus the biggest issues are with the fundamental architecture. So yeah, it's pretty much impossible to make distutils better.
Agreed. I was more thinking about obviously correct 1 line patches that go unmerged. Ralf
Instead, the general goal of the packaging maintainers at this point is to get rid of distutils. The big thing that will enable this is PEP 517, which replaces setup.py with a much smaller and properly-specified interface for new build systems to implement. The hope is that if creating alternative build systems becomes an easy and supported thing to do, then better options will emerge, and we'll get out of this situation where there's one tool everyone has to use and which isn't really good for anyone.
Unfortunately, support for this hasn't quite landed in pip yet. But that's the direction that things are going.
-n
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year. Hi! As a potential contributor to SciPy I would say that one of the things I’m most excited about, and one of the things that has kept me from getting too deeply involved in the C/C++ parts of NumPy/SciPy is the lack of proper IDE support. CMake solves this to an extent. I’ve found myself to be relatively comfortable with projects that used CMake rather than (for example) GNU Make or Autotools for the same reason. Best Regards, Hameer Abbasi Sent from Astro <https://www.helloastro.com> for Mac
It looks like you have to have CMake installed though (http://scikit-build.readthedocs.io/en/latest/installation.html). This is a hurdle in itself.
The docs are a little out-of-date. This patch will bring them up to speed: https://github.com/scikit-build/scikit-build/pull/339 With pip 10, there is no need to worry about installing CMake since there is a cmake PyPI package and the dependency can be added to pyproject.toml.
There are binary wheels (https://pypi.org/project/cmake) and it's packaged for conda, plus good binary installers - for a development setup that's about as easy as it is going to get.
Yes, and quite a few folks are successfully using the cmake PyPI package, over 8,000 a month according to current stats: https://pypistats.org/packages/cmake
Can you (and maybe Ralf) be more specific --- in what way there are problems?
What advantages scikit-build (which glues CMake and setuptools together) brings?
A good summary is given in the recent SciPy 2018 Conference talk by @jcfr (in CC): https://www.youtube.com/watch?v=QVkg-cC5oe4
Of course, distutils forces you to do CFLAGS/etc customization via environment variables, and BLAS/LAPACK selection via site.cfg which is not so convenient.
Yes -- scikit-build supports CFLAGS, etc. environmental variables but you can also pass flags and build configuration options more generally and more explicitly in the `python setup.py [...]` call. CMake has good discovery of BLAS/LAPACK, but you can also easily and robustly override what would be discovered through system introspection. Reference LAPACK uses CMake for its build system, so it fits in nicely.
Related: scikit-build seems to have quite a bit of momentum. I've never used CMake before, and while I know opinions on CMake aren't 100% positive, scikit-build looks like a significant improvement over distutils. I think we should try to integrate it at some point in the coming year.
Hi! As a potential contributor to SciPy I would say that one of the things I’m most excited about, and one of the things that has kept me from getting too deeply involved in the C/C++ parts of NumPy/SciPy is the lack of proper IDE support. CMake solves this to an extent. I’ve found myself to be relatively comfortable with projects that used CMake rather than (for example) GNU Make or Autotools for the same reason.
Yes, with CMake you can generate a build system or IDE of preference, i.e. Visual Studio, XCode, Sublime Text, Eclipse, various Makefile's, Ninja files, ... The full list can be found here: https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html Hope this helps, Matt
Hi CJ, I don't think that the issues addressed by bento have been tackled much by anything in the python ecosystem. It is true that binary distribution today is much better than it was in 2008/2009 when I started working on Bento, which means much fewer people need to deal w/ distutils errors on a day to day basis. But distutils is pretty much as bad today as it was 10 years ago. If I were to start it today, I would have made different choices, as I was still very much a novice when I started bento :) But IMO the main failure is the lack of community around it. I still think the main ideas around it is where distutils2/whatever-it-is-called should go. David On Fri, Mar 31, 2017 at 4:55 PM, CJ Carey <perimosocordiae@gmail.com> wrote:
With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento.
The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems.
If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (10)
-
Andrew Nelson -
CJ Carey -
David Cournapeau -
Evgeni Burovski -
Hameer Abbasi -
Matthew McCormick -
Nathaniel Smith -
Pauli Virtanen -
Ralf Gommers -
Stefan van der Walt