[Python-Dev] Making PyInterpreterState an opaque type

[INADA Naoki]

On Thu, Feb 21, 2019 at 9:23 PM Paul Moore <p.f.moore at gmail.com> wrote:
>
> On Thu, 21 Feb 2019 at 12:12, Antoine Pitrou <antoine at python.org> wrote:
>
> > Actually, it would be interesting to have some kind of survey of C
> > extensions (through random sampling? or popularity?) to find out why the
> > developers had to write a C extension in the first place and what their
> > concerns are.
>
> Indeed. There's also embedding, where I suspect there's a much higher
> likelihood that performance isn't key.
>
> And in your survey, I'd split out "needs the Python/C interface to be
> fast" from "needs internal operations to be fast, but data transfer
> between C and Python isn't as critical". I suspect there's a lot of
> people who believe they are in the former category, but are actually
> in the latter...
>
> Paul

As my experience, I can't believe the majority of extensions are in category B.

And when speed is matter, PyPy support is not mandatory.  PyPy is fast enough
with pure Python implementation.  Python/C API doesn't make PyPy fast when
creating or reading massive objects.

These are my recent Python/C API usages.

# msgpack

msgpack is a serialization format like JSON, but in binary format.

It contains pure Python implementation and Cython implementation.
Cython implementation is for CPython.  PyPy is fast enough with pure
Python implementation.

It is important to fast object access.  It is like marshal in Python.
So this is clearly in category A.


# mysqlclient

It is binding of libmysqlclient or libmariadbclient.
In some case, query can return many small data and I need to convert them
to Python object.
So this library is basically in category B, but sometime in C.


# MarkupSafe

It is library which escape strings.

It should handle massive small string chunks, because it is called
from template engine.

So this library is in category A.
Note that MarkupSafe having pure Python version, like msgpack.
While MarkupSafe is not optimized for PyPy yet, it's possible to add
implementation for PyPy written in Python using PyPy's string builder [1].

[1] https://morepypy.blogspot.com/2011/10/speeding-up-json-encoding-in-pypy.html

-- 
INADA Naoki  <songofacandy at gmail.com>