Static checker for cython extension dependencies?
Hi all, I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython. I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport. To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure. https://bitbucket.org/yt_analysis/yt/pull-requests/2220 I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development. Thanks for your help! -Nathan
You should be using cythonize rather than listing and maintaining the Extension definitions themselves. http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60 Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file? FWIW, our setup.py file is here: https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available. On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here: https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies or depend directly on C code. For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object. I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file. That said, I'd love to hear alternate ideas. On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure
other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... that that
adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies
meaning? In my cursory glance, these all look fine, but perhaps you could point me at a case that isn't very nice. Also, cythonize isn't necessarily all-or-nothing--you can always manually add Extensions to your list.
or depend directly on C code.
cythonize handles dependencies on C code just fine
For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
The embedded metadata is the union of what was in the Extension object and the extra dependencies it found.
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file.
That said, I'd love to hear alternate ideas.
"Parse pyx files for cimports" (recursively, handing includes and cdef extern from ...) is essentially re-implementing cythonize. What is the advantage of maintaining them by hand? Cythonize was invented because people are really bad at maintaining these lists manually...and they can be entirely deduced.
On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us...
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
Hmm, so I've looked into this a bit further, and it looks like the
On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote: metadata
isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies
meaning? In my cursory glance, these all look fine, but perhaps you could point me at a case that isn't very nice. Also, cythonize isn't necessarily all-or-nothing--you can always manually add Extensions to your list.
or depend directly on C code.
cythonize handles dependencies on C code just fine
For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
The embedded metadata is the union of what was in the Extension object and the extra dependencies it found.
Hmm, maybe I made a mistake, but when I passed an Extension object to cythonize earlier, I found that the metadata that got written out to the autogenerated C only included the dependencies that were listed in the Extension object's declaration, not additional ones inferred by cython. Perhaps this is a bug?
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file.
That said, I'd love to hear alternate ideas.
"Parse pyx files for cimports" (recursively, handing includes and cdef extern from ...) is essentially re-implementing cythonize. What is the advantage of maintaining them by hand? Cythonize was invented because people are really bad at maintaining these lists manually...and they can be entirely deduced.
The advantage here is that in my couple hours of playing with using cythonize for our setup.py script, I couldn't get it to correctly generate the metadata I needed to validate the dependencies, so it wasn't providing any added value. Again, it's entirely possible I was making a mistake somewhere. Also, separately, the fake_cythonize script you suggested will error out if cython isn't installed and the cython-generated C files do not exist yet. That certainly makes sense if you expect developers to have cython available, but it's a downgrade with respect to our current setup. Right now we have it set up so that setuptools will install cython into the build environment while it is processing C extensions if it is not already installed. This means if new developers or users come along and want to build yt from source, they won't need to explicitly install cython first - setuptools will take care of it for us. This makes our installation instructions simpler and causes fewer head-scratching crashes when people try to compile yt from source.
On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com
wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us...
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for
the
first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
On Fri, Jun 10, 2016 at 3:21 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies
meaning? In my cursory glance, these all look fine, but perhaps you could point me at a case that isn't very nice. Also, cythonize isn't necessarily all-or-nothing--you can always manually add Extensions to your list.
or depend directly on C code.
cythonize handles dependencies on C code just fine
For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
The embedded metadata is the union of what was in the Extension object and the extra dependencies it found.
Hmm, maybe I made a mistake, but when I passed an Extension object to cythonize earlier, I found that the metadata that got written out to the autogenerated C only included the dependencies that were listed in the Extension object's declaration, not additional ones inferred by cython. Perhaps this is a bug?
Sounds like a bug to me.
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file.
That said, I'd love to hear alternate ideas.
"Parse pyx files for cimports" (recursively, handing includes and cdef extern from ...) is essentially re-implementing cythonize. What is the advantage of maintaining them by hand? Cythonize was invented because people are really bad at maintaining these lists manually...and they can be entirely deduced.
The advantage here is that in my couple hours of playing with using cythonize for our setup.py script, I couldn't get it to correctly generate the metadata I needed to validate the dependencies, so it wasn't providing any added value. Again, it's entirely possible I was making a mistake somewhere.
:(. I'll take a quick stab at it.
Also, separately, the fake_cythonize script you suggested will error out if cython isn't installed and the cython-generated C files do not exist yet.
The fake_cythonize script (and metadata) is intended for the case that the user doesn't have Cython, but the C files are shipped. Sounds like you require Cython (installing it for the user if they don't have it).
That certainly makes sense if you expect developers to have cython available, but it's a downgrade with respect to our current setup. Right now we have it set up so that setuptools will install cython into the build environment while it is processing C extensions if it is not already installed. This means if new developers or users come along and want to build yt from source, they won't need to explicitly install cython first - setuptools will take care of it for us. This makes our installation instructions simpler and causes fewer head-scratching crashes when people try to compile yt from source.
Makes sense. In this case, you would have to avoid importing and calling cythonize until after your setup.py invoked setup.
On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us...
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote: > Hi all, > > I'm working on a pretty large python/cython codebase (yt) that has > many > interdependent C extensions written in cython. > > I've found it to be pretty hit or miss to depend on contributors > to > manually > update cython dependency information in our setup.py file. The > dependency > information seems to only be used by setuptools to trigger > recompilation, > but critically setuptools will happily compile a C extension for > the > first > time if the depends information is incomplete. This means that > during > development, if I update a cython routine, there's no way to > ensure > that > other cython routines that cimport the one I modified will be > recompiled > unless I manually ensure the depends information is updated > whenever > cython > code gains or loses a cimport. > > To make that more concrete, here's a pull request I just made to > yt > that > adds missing dependencies for a cython header. Without this pull > request, > setuptools fails to recompile these routines when > selection_routines.pxd > changes, causing a build failure. > > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 > > I think it should be possible to write a test for this by defining > the > dependency information outside of setup.py and parsing grep and > looking > for > all cython files that cimport other cython files defined inside > yt. > However, > before I do that, I'm curious whether anyone has done something > similar, > or > if there is some other way of forcing the dependency information > to > be > complete on the first compilation, rather than just for subsequent > incremental recompilations during development. > > Thanks for your help! > > -Nathan > > _______________________________________________ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
https://bitbucket.org/yt_analysis/yt/pull-requests/2225/use-cythonize-to-man... On Fri, Jun 10, 2016 at 3:45 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Jun 10, 2016 at 3:21 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies
meaning? In my cursory glance, these all look fine, but perhaps you could point me at a case that isn't very nice. Also, cythonize isn't necessarily all-or-nothing--you can always manually add Extensions to your list.
or depend directly on C code.
cythonize handles dependencies on C code just fine
For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
The embedded metadata is the union of what was in the Extension object and the extra dependencies it found.
Hmm, maybe I made a mistake, but when I passed an Extension object to cythonize earlier, I found that the metadata that got written out to the autogenerated C only included the dependencies that were listed in the Extension object's declaration, not additional ones inferred by cython. Perhaps this is a bug?
Sounds like a bug to me.
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file.
That said, I'd love to hear alternate ideas.
"Parse pyx files for cimports" (recursively, handing includes and cdef extern from ...) is essentially re-implementing cythonize. What is the advantage of maintaining them by hand? Cythonize was invented because people are really bad at maintaining these lists manually...and they can be entirely deduced.
The advantage here is that in my couple hours of playing with using cythonize for our setup.py script, I couldn't get it to correctly generate the metadata I needed to validate the dependencies, so it wasn't providing any added value. Again, it's entirely possible I was making a mistake somewhere.
:(. I'll take a quick stab at it.
Also, separately, the fake_cythonize script you suggested will error out if cython isn't installed and the cython-generated C files do not exist yet.
The fake_cythonize script (and metadata) is intended for the case that the user doesn't have Cython, but the C files are shipped. Sounds like you require Cython (installing it for the user if they don't have it).
That certainly makes sense if you expect developers to have cython available, but it's a downgrade with respect to our current setup. Right now we have it set up so that setuptools will install cython into the build environment while it is processing C extensions if it is not already installed. This means if new developers or users come along and want to build yt from source, they won't need to explicitly install cython first - setuptools will take care of it for us. This makes our installation instructions simpler and causes fewer head-scratching crashes when people try to compile yt from source.
Makes sense. In this case, you would have to avoid importing and calling cythonize until after your setup.py invoked setup.
On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote: > > You should be using cythonize rather than listing and maintaining > the > Extension definitions themselves. > > > > > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... > > > https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing > > On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum > <nathan12343@gmail.com> > wrote: > > Hi all, > > > > I'm working on a pretty large python/cython codebase (yt) that has > > many > > interdependent C extensions written in cython. > > > > I've found it to be pretty hit or miss to depend on contributors > > to > > manually > > update cython dependency information in our setup.py file. The > > dependency > > information seems to only be used by setuptools to trigger > > recompilation, > > but critically setuptools will happily compile a C extension for > > the > > first > > time if the depends information is incomplete. This means that > > during > > development, if I update a cython routine, there's no way to > > ensure > > that > > other cython routines that cimport the one I modified will be > > recompiled > > unless I manually ensure the depends information is updated > > whenever > > cython > > code gains or loses a cimport. > > > > To make that more concrete, here's a pull request I just made to > > yt > > that > > adds missing dependencies for a cython header. Without this pull > > request, > > setuptools fails to recompile these routines when > > selection_routines.pxd > > changes, causing a build failure. > > > > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 > > > > I think it should be possible to write a test for this by defining > > the > > dependency information outside of setup.py and parsing grep and > > looking > > for > > all cython files that cimport other cython files defined inside > > yt. > > However, > > before I do that, I'm curious whether anyone has done something > > similar, > > or > > if there is some other way of forcing the dependency information > > to > > be > > complete on the first compilation, rather than just for subsequent > > incremental recompilations during development. > > > > Thanks for your help! > > > > -Nathan > > > > _______________________________________________ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
It's not clear to me what you're doing, but it all sounds too complicated to me. I have Cython extensions that depend on C++ code, and I don't have to resort to these kinds of contortions. One model that works well is to build your C code as a shared library. Make each Cython module in a single file, and have it depend on your C library. In my case, I don't use any of the Python/Cython build systems; I just call cythonize to get C++ code, and then build that C++ code as I would build any other C++ code (in CMake). There's then a bit of CMake/Python stuff needed to get CMake to install in the correct Python locations. Another way would be to build your supporting C code as a separate package (with CMake or Autotools, etc.) --- and then use the Python build system for the actual Cython extension. Now you have two packages instead of one, but that doesn't bother me much. I chose the all-CMake way because I put a lot of time and effort into learning CMake, and didn't want to have to put more time and effort into learning another build system, just so I could build Python. But either way should work. -- Elizabeth On Fri, Jun 10, 2016 at 4:18 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies or depend directly on C code. For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object.
I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file.
That said, I'd love to hear alternate ideas.
On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
We write metadata in the generated C files for just this reason. You can fall back to something like https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if cython is not available.
On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60
Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file?
FWIW, our setup.py file is here:
https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb@gmail.com> wrote:
You should be using cythonize rather than listing and maintaining the Extension definitions themselves.
https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Hi all,
I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython.
I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure
other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport.
To make that more concrete, here's a pull request I just made to yt
adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure.
https://bitbucket.org/yt_analysis/yt/pull-requests/2220
I think it should be possible to write a test for this by defining
http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-us... that that the
dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development.
Thanks for your help!
-Nathan
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
_______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
participants (3)
-
Elizabeth A. Fischer -
Nathan Goldbaum -
Robert Bradshaw