[Python-ideas] Re: New Tool Proposal

[anthony.flury]

On 10/05/2022 15:04, Dan Stromberg wrote:
>
> On Tue, May 10, 2022 at 3:15 AM Chris Angelico <rosuav at gmail.com> wrote:
>
>     > It is often the case that developer write Code in Python and
>     then convert to a C extension module for performance regions.
>     >
>     > A C extension module has a lot of boiler plate code - for
>     instance the Structures required for each class, the functions for
>     Module initialization etc.
>     >
>     > My Idea is a simple tool that uses introspection tools to take a
>     Python module and to generate the relevant boiler plate for the
>     module - including blank functions for the module classes and for
>     methods. This tool would use type annotations (if given) to make
>     sensible choices for parameter and attribute types, including
>     using int and float directly rather than Internal objects
>     (depending on tool options).
>
>
> Two things to say about this:
> 1) Sometimes abandoning a pure python module for a C extension for 
> performance is a mistake - because Pypy is probably going to be much 
> faster with the pure python module

Dan,

Thanks for your response, but I think PyPy will have a long way to go to 
make JIT generated code more efficient than hand crafted C extension.

for instance PyPy will almost certainly be unable to determine if 
integers, floats etc will need to use the Python runtime or can be 
optimized to use direct C types, another example would be lists - there 
are some cases where the C extension will always be a lot more efficient 
using C arrays than using the C list runtime machinery.

While there might be some cases where PyPy will be more efficient, I 
don't think that will be the case for all programs, and that ignores the 
fact that PyPi has major issues with the CAPI.


> 2) I've had some luck using m4 to maintain a single source file that 
> is used to automatically generate both pure python and cython.  This 
> is a little like using cpp in a C project.
>
> For examples of #2, perhaps see:
> https://stromberg.dnsalias.org/~strombrg/treap/
> https://stromberg.dnsalias.org/svn/rolling_checksum_mod/trunk/
> https://stromberg.dnsalias.org/~strombrg/sort-comparison/
>
> It's often nice to keep the lines of the pure-python and cython having 
> a 1-1 relationship, so that tracebacks report useful line numbers 
> either way.  However, in the treap example I've dispensed with that 
> because some methods were almost identical but had some boilerplate - 
> and m4 was able to handle that nicely at the cost of lines being 1-1.
>
> HTH
-- 
Anthony Flury
*Moble*: +44 07743 282707
*Home*: +44 (0)1206 391294
*email*: anthony.flury at btinternet.com