[Cython] Utilities, cython.h, libcython

[Robert Bradshaw]

On Wednesday, October 5, 2011, mark florisson wrote:

> On 5 October 2011 01:46, Robert Bradshaw <robertwb at math.washington.edu<javascript:;>>
> wrote:
> > On Tue, Oct 4, 2011 at 2:19 PM, mark florisson
> > <markflorisson88 at gmail.com <javascript:;>> wrote:
> >> Hey,
> >>
> >> I briefly mentioned something about this in a pull request, but maybe
> >> it deserves some actual discussion on the ML.
> >>
> >> So I propose that after fused types gets merged we try to move as many
> >> utility codes as possible to their utility code files (unless they are
> >> used in pending pull requests or other branches). Preferably this will
> >> be done in one or a few commits. How should we split up the work, any
> >> volunteers? Perhaps people who wrote certain utilities also want to
> >> move them? In that case, we should start a new branch and then merge
> >> that into master when it's done.
> >> We could actually move things before fused types get merged, as long
> >> as we don't touch binding_cfunc_utility_code.
> >
> > +1 to moving towards this, but I don't see the urgency or need to do
> > it all at once (though if there's going to be a big push, lets
> > coordinate on a wiki or trac).
>
> Hm, perhaps there is no strict need to hurry, as long as we take care
> not to modify utilities after they have been moved. The wiki could be
> great for that, but I personally don't keep track of everyone's
> branches, so I don't know which utility is modified by whom (if at
> all), so strictly speaking (to avoid painful merges) I'd have to ask
> everyone each time I wanted to move something, or dig through
> everyone's branches.
>

I was proposing that everyone lists the utility code sections that are
likely to cause merge conflicts on a wiki page, and the rest are fair game.


> >> Before we go there, Stefan, do we still want to implement the header
> >> .ini style which can list dependencies and such? I personally don't
> >> care very much about it, but memoryviews and the utility loaders are
> >> merged so if someone wants to take up that job, it'd be good to do
> >> before moving the utilities.
> >>
> >> Another issue is that Cython compile time is increasing with the
> >> addition of control flow and cython utilities. If you use fused types
> >> you're also going to combinatorially add more compile time.
> >
> > Yeah, this was especially obvious with, e.g. cython.compile(...). (In
> > particular, some utility code was being parsed before it could even
> > figure out whether it needed to do a full re-compile...)
> >
> >> I'm sure
> >> this came up earlier, but I really think we should have a libcython
> >> and a cython.h. libcython (a shared library) should contain any common
> >> Cython-specific code not meant to be inlined, and cython.h any types,
> >> macros and inline functions etc. This will decrease Cython and C
> >> compile time, and will also make executables smaller.
> >
> > +1. Yes, we talked about this earlier, but nothing concrete was
> > planned. It's probably worth a CEP, if anything to have a concrete
> > plan recorded somewhere other than a series of mailing list threads
> > (though discussion tends to work best here).
> >
> >> This could be
> >> enabled using a command line option to Cython, as well as with
> >> distutils, eventually we may decide to make it the default (lets
> >> figure that out later). Preferably libcython.so would be installed
> >> alongside libpython.so and cython.h inside the Python include
> >> directory. Assuming multiple versions of Cython and multiple Python
> >> installations, we'd need to come up with a versioning scheme for
> >> either.
> >
> > I would propose a cython.h file that sits in Cython/Compiler/Include
> > (or similar), as a first step. The .pyx -> .c pass could be configured
> > to copy this to a specific location (for shipping just the generated
> > .c files).
>
> That would be fine as well. It might be convenient for users in that
> case if we could provide a cython.get_include() in addition to the
> distutils hooks, and a cython-config script.
>

For sure. We could also have a cython.get_shared_library() (common_code?
cython_module?) which would return an Extension object to build.


> > One option is to build the shared library as a companion
> > _cython_x_y_z.so module which, while not as efficient as linking at
> > the C level, would probably be much simpler to implement in a
> > cross-platform way. (This perhaps merits some benchmarks, but the main
> > contents is likely to be things like shared classes and objects.)
> > Actually linking .so files from modules that cimport each other would
> > be a nice feature down the road anyways. Again, the associated .c file
> > could be (optionally) generated/copied during the .pyx -> .c step.
> > Installation would determine if the required module exists, and if not
> > build and install it.
>
> Hm, that's a really good idea. I think the only overhead would be the
> capsule unpacking and pointer duplication, but that shouldn't suddenly
> be an issue. That means we don't have to do any versioning of the
> libraries and the symbols to avoid clashes in a flat namespaces as
> Stefan mentioned.
>

I'm not sure what the overhead is, if any, in calling function pointers vs.
actually linking things together at the C level (which is essentially the
same idea, but perhaps addresses are resolved at library load time rather
than requiring a dereference on each call?)


> >> We could also provide a static library there, for users who want to
> >> link and ship a compiled and statically linked version of their code.
> >> For a local Cython that isn't built, we can ignore the header and
> >> shared library option and issue a warning or some such.
> >>
> >> Lastly, I think we also should figure out a way to serialize Entry
> >> objects from CythonUtilities, which could easily and swiftly be loaded
> >> when creating the cython scope. It's quite a pain to declare all
> >> entries for utilities you write manually, so what I mostly did was
> >> parse the utility up to and including AnalyseDeclarationsTransform,
> >> and then retrieve the entries from there.
> >
> > This would be really nice too. Way back in the day I did some work
> > with trying to pickle full module scopes, but that soon became too
> > painful as there are so many far-reaching references. Pickling
> > individual Entries and re-building modules will probably be a more
> > tractable goal. Eventually, I'd like to see a way to cache the full
> > pxd pipeline.
> >
> > - Robert
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org <javascript:;>
> > http://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org <javascript:;>
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20111005/fef12bbb/attachment-0001.html>