[Cython] Static checker for cython extension dependencies?

Nathan Goldbaum nathan12343 at gmail.com
Fri Jun 10 18:21:57 EDT 2016


On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > Hmm, so I've looked into this a bit further, and it looks like the
> metadata
> > isn't going to be useful for us. Many of our extensions can't be
> > autogenerated by cython because they have non-trivial dependencies
>
> meaning? In my cursory glance, these all look fine, but perhaps you
> could point me at a case that isn't very nice. Also, cythonize isn't
> necessarily all-or-nothing--you can always manually add Extensions to
> your list.


> > or depend directly on C code.
>
> cythonize handles dependencies on C code just fine
>
> > For these extensions, cythonize must be passed an
> > instantiated Extension object, and the metadata embedded in the compiled
> C
> > file just uses whatever metadata is attached to the Extension object.
>
> The embedded metadata is the union of what was in the Extension object
> and the extra dependencies it found.
>

Hmm, maybe I made a mistake, but when I passed an Extension object to
cythonize earlier, I found that the metadata that got written out to the
autogenerated C only included the dependencies that were listed in the
Extension object's declaration, not additional ones inferred by cython.
Perhaps this is a bug?


>
> > I think my original idea - to parse pyx files for cimports and add a
> test -
> > should allow us to manage this more safely in the future while still
> > maintaining fine-grained control over dependencies of C extensions in our
> > setup.py file.
> >
> > That said, I'd love to hear alternate ideas.
>
> "Parse pyx files for cimports" (recursively, handing includes and cdef
> extern from ...) is essentially re-implementing cythonize. What is the
> advantage of maintaining them by hand? Cythonize was invented because
> people are really bad at maintaining these lists manually...and they
> can be entirely deduced.


The advantage here is that in my couple hours of playing with using
cythonize for our setup.py script, I couldn't get it to correctly generate
the metadata I needed to validate the dependencies, so it wasn't providing
any added value. Again, it's entirely possible I was making a mistake
somewhere.

Also, separately, the fake_cythonize script you suggested will error out if
cython isn't installed and the cython-generated C files do not exist yet.
That certainly makes sense if you expect developers to have cython
available, but it's a downgrade with respect to our current setup. Right
now we have it set up so that setuptools will install cython into the build
environment while it is processing C extensions if it is not already
installed. This means if new developers or users come along and want to
build yt from source, they won't need to explicitly install cython first -
setuptools will take care of it for us. This makes our installation
instructions simpler and causes fewer head-scratching crashes when people
try to compile yt from source.


>
> > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >> We write metadata in the generated C files for just this reason. You
> >> can fall back to something like
> >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
> >> cython is not available.
> >>
> >>
> >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <
> nathan12343 at gmail.com>
> >> wrote:
> >> > The reason we haven't done that is we would like our setup.py script
> to
> >> > be
> >> > runnable without cython being installed. I think cythonize is being
> >> > invoked
> >> > (or something similar to it) by setuptools, using a feature added in
> >> > setuptools 18.0:
> >> > https://setuptools.readthedocs.io/en/latest/history.html#id60
> >> >
> >> > Is there a way to use cythonize for this build workflow without
> >> > importing it
> >> > at the top-level in our setup.py file?
> >> >
> >> > FWIW, our setup.py file is here:
> >> >
> >> >
> https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
> >> >
> >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com
> >
> >> > wrote:
> >> >>
> >> >> You should be using cythonize rather than listing and maintaining the
> >> >> Extension definitions themselves.
> >> >>
> >> >>
> >> >>
> >> >>
> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
> >> >>
> >> >>
> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
> >> >>
> >> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
> >> >> <nathan12343 at gmail.com>
> >> >> wrote:
> >> >> > Hi all,
> >> >> >
> >> >> > I'm working on a pretty large python/cython codebase (yt) that has
> >> >> > many
> >> >> > interdependent C extensions written in cython.
> >> >> >
> >> >> > I've found it to be pretty hit or miss to depend on contributors to
> >> >> > manually
> >> >> > update cython dependency information in our setup.py file. The
> >> >> > dependency
> >> >> > information seems to only be used by setuptools to trigger
> >> >> > recompilation,
> >> >> > but critically setuptools will happily compile a C extension for
> the
> >> >> > first
> >> >> > time if the depends information is incomplete. This means that
> during
> >> >> > development, if I update a cython routine, there's no way to ensure
> >> >> > that
> >> >> > other cython routines that cimport the one I modified will be
> >> >> > recompiled
> >> >> > unless I manually ensure the depends information is updated
> whenever
> >> >> > cython
> >> >> > code gains or loses a cimport.
> >> >> >
> >> >> > To make that more concrete, here's a pull request I just made to yt
> >> >> > that
> >> >> > adds missing dependencies for a cython header. Without this pull
> >> >> > request,
> >> >> > setuptools fails to recompile these routines when
> >> >> > selection_routines.pxd
> >> >> > changes, causing a build failure.
> >> >> >
> >> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
> >> >> >
> >> >> > I think it should be possible to write a test for this by defining
> >> >> > the
> >> >> > dependency information outside of setup.py and parsing grep and
> >> >> > looking
> >> >> > for
> >> >> > all cython files that cimport other cython files defined inside yt.
> >> >> > However,
> >> >> > before I do that, I'm curious whether anyone has done something
> >> >> > similar,
> >> >> > or
> >> >> > if there is some other way of forcing the dependency information to
> >> >> > be
> >> >> > complete on the first compilation, rather than just for subsequent
> >> >> > incremental recompilations during development.
> >> >> >
> >> >> > Thanks for your help!
> >> >> >
> >> >> > -Nathan
> >> >> >
> >> >> > _______________________________________________
> >> >> > cython-devel mailing list
> >> >> > cython-devel at python.org
> >> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >> >
> >> >> _______________________________________________
> >> >> cython-devel mailing list
> >> >> cython-devel at python.org
> >> >> https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > cython-devel mailing list
> >> > cython-devel at python.org
> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160610/9ce24552/attachment.html>


More information about the cython-devel mailing list