[pypy-commit] cffi cffi-1.0: The non-private part of the accepted grant proposal

[arigo]

Author: Armin Rigo <arigo at tunes.org>
Branch: cffi-1.0
Changeset: r1690:16dc660b4ee7
Date: 2015-04-10 19:16 +0200
http://bitbucket.org/cffi/cffi/changeset/16dc660b4ee7/

Log:	The non-private part of the accepted grant proposal

diff --git a/new/cffi-1.0.rst b/new/cffi-1.0.rst
new file mode 100644
--- /dev/null
+++ b/new/cffi-1.0.rst
@@ -0,0 +1,124 @@
+
+===========================
+Grant Proposal for CFFI 1.0
+===========================
+
+*Accepted by the PSF board on April 4, 2015*
+
+This Grant Proposal is to give a boost towards "CFFI 1.0".  Two main
+issues with the current CFFI need to be solved: the difficulties of
+installation, and the potentially large time taken at import.
+
+1. The difficulties of installation can be seen from outside by looking
+at various workarounds and 3rd-party documentation that have grown into
+existence.  For example, the `setup.py` of projects like cryptography,
+PyNaCl and bcrypt deploys workarounds that are explicitly documented in
+https://caremad.io/2014/11/distributing-a-cffi-project/.
+
+2. The time taken at import is excessive in some cases.  For example,
+importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order
+of 10 to 20 seconds (and this is the "fast" case where the compiler
+doesn't need to be invoked any more).
+
+
+Technical Overview
+------------------
+
+"CFFI" is an existing Python project which complements the ctypes,
+SWIG and Cython approaches to ease writing C Extension Modules for
+Python.  It has several advantages over the previous approaches, which
+are presented at the start of the documentation at
+http://cffi.readthedocs.org/en/latest/ .  It has been very successful
+so far: http://pypi-ranking.info/alltime records almost 7 million
+downloads (for comparison, the #1 of all packages has almost 36
+million downloads).  CFFI works on any Python >= 2.6, including 3.x,
+as well as on PyPy.
+
+One problem is that while getting started with CFFI is very easy, the
+installation process of a package that uses CFFI has got its rough
+edges.  CFFI (at least in its "verify()" mode) is based on calling the
+C compiler to get information about the exact C types, structures,
+argument types to functions, and so on.  The C compiler is invoked
+transparently at run-time, and the results cached.  A
+correctly-installed package using CFFI should cache the results at
+installation time, but it can be difficult to ensure that no more
+run-time compiler invocation is needed; doing so requires following
+some extra guidelines or understanding some internal details.  (The
+problem is particularly acute on Windows where a typical user might
+not have a proper C compiler installed.)
+
+To fix this, we have in mind adding a different CFFI mode (replacing
+"verify()"), while keeping the access to the underlying C library
+unmodified.  In this mode, the code containing the cdef() and verify()
+invocations would be moved to a separate Python source file.  Running
+that Python file would produce a dynamically-linked library.  There
+would be no caching logic involved; you would need to run it
+explicitly during development whenever you made changes to it, to
+re-generate and re-compile the dynamically-linked library.
+
+When distributed, the same file would be run (once) during
+installation.  This can be fully automated in setuptools-based
+setup.py files; alternatively, it can be done in distutils-based
+setup.py files by requiring prior manual installation of CFFI itself.
+
+A major difference with the existing verify() approach would be that
+the ``.so/.dll/.dylib`` file would not be immediately loaded into the
+process; you would load it only from the installed program at
+run-time, and get the ``ffi`` and ``lib`` objects in this way (these
+are the two objects that you use so far to access a C library with
+verify()).
+
+Additionally, this would solve another issue: every import of a large
+CFFI-using package takes a while so far.  This is caused by CFFI
+needing to parse again the C source code given in the cdef() (adding a
+run-time dependency to the ``pycparser`` and ``ply`` packages).  CFFI
+also computes a CRC to know if it can reuse its cache.  In the
+proposed change, all the cdef() code would be pre-parsed and stored in
+the dynamically-linked library, and no CRC would be needed.  This
+would massively reduce the import times.
+
+
+Grant objective
+---------------
+
+The objective is to give a boost towards "CFFI 1.0", which needs to have
+the functionalities described above in order to solve the two main
+issues with the current CFFI: the difficulties of installation, and the
+time taken at import.
+
+Included in the objective: the internal refactorings of CFFI that are
+needed to get it done cleanly.  The goal is to avoid simply adding
+another layer on top of the old unchanged CFFI.
+
+This work may happen eventually in any case, but support from the PSF
+would help make it happen sooner rather than later.
+
+
+Grant size
+----------
+
+2'500 US$ for supporting the development time.  This would cover 2.5
+weeks of full-time work at the part-time cost of 25 US$ per hour.
+
+The estimated work time until the CFFI 1.0 release is a bit larger
+than that (I estimate it at roughly 4 weeks), but 2.5 weeks should
+cover all the basics.  An extended grant size of 4'000 US$ would be
+appreciated but not required ``:-)``
+
+
+Grant beneficiaries
+-------------------
+
+Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time
+work.
+
+
+Grant follow-up
+---------------
+
+I will report on the success of the grant on the CFFI mailing list and
+on the blog I usually post to (the PyPy blog) and mention the PSF as
+providing the grant.  The PSF will receive an email pointing to these
+postings once they are out.  Moreover a full CFFI 1.0 release should
+follow (likely starting with beta versions); the PSF will receive
+another email pointing to it.