On Sun, Apr 23, 2023, 3:43 PM turnbull <turnbull@sk.tsukuba.ac.jp> wrote:
This isn't a PEP yet: it's a set of requirements.  A PEP eventually
needs to
say how to implement the requirements, and even at this "proto-PEP"
it needs to be plausible that it's implementable.  It's on you to
how your very ambitious requirements can be satisfied in Python. 
going to ask you for an implementation, but references to related tech
HPy and ctypes combined with discussion of how they do or don't meet
requirements would be helpful.

2023-04-23 21:13 に Evan Greenup via Python-ideas さんは書きました:

> However there is a lot of limitation in those library [like PyO3].

For good reason.  Python does not share data structures with those other
languages.  CPython's native data structures are a subset of those
in C -- obviously, because CPython is implemented in C.  But other
will implement "mutable extensible memory-safe sequence" (ie, Python's
in different ways (and famously C doesn't provide that!)  On the other
data structures in other languages may have no built-in equivalent in
The low-level cytpes stdlib module provides the flexibility you want,
but it's
implementation-dependent on the foreign side, and must be.

> * The project structure is rigid.

This is a complaint about specific third-party libraries that provide
level wrapping of a fundamentally low-level facility.  I would *expect*
project structure to be rigid.

I think you might find it easier to present the proposal convincingly if
"build up" from ctypes, instead of "building down" from PyO3.

> * It is really ridiculous when you want to stick some item on the
> wall. You need to totally redesign this item and manufacture a new
> item to fit the glue you are going to use.

This is not true of ctypes, which is designed as the thinnest possible
wrapper around other languages (specifically C, but to the extent that
most C
implementations provide facilities for calling FORTRAN and other such
languages, it should be possible to extend ctypes to those languages in

> As a glue language, Python should be designed to glue other native
> programming
> language as a feature of Python programming language itself

As far as I can see this is not feasible, and vastly overemphasizes
Python's role
as a glue language.  Python is a programming language first, and the
business of
the Python programming language is to be Python.  Interfacing to other
is going to be more or less painful depending on how closely the
internals are
related, and in general it will be hardware-dependent.

> * The interface is universal across all variant and version of Python
> implementation

It took Microsoft 15 versions and a couple of decades to manage this
with just
its own runtime library.  Remember, not only do Python internals change
version to version, but so do those of other languages.  C++ is infamous
incompatibility, in fact.  It's hard to imagine that the Python side of
interface can be completely independent of the other language, when the
point is that the other language has specific features that *Python does

As Jelle mentions, a standardized ABI for Python is in process, the
iteration being the HPy project.  However AIUI the goal is a consistent
across Python versions, not making construction of FFIs easier.  All it
do I believe is remove ABI compatibility across Python versions from the
set of
problems an FFI needs to deal with.  That's useful, but doesn't remove
any of
the complexity caused by different representations in the target

> It is a lively data structure with in-memory representation, they are
> unified
> no matter what Python variant is used and what low level native
> language is
> used.

So you're suggesting an intermediate data representation, likely
requiring two
translations (Python to intermediate and intermediate to target
language) each
time data is to be transferred from Python to a target language and
back.  If
HPy succeeds then that ABI can be frozen as both the Python ABI and the
mediate representation, of course.

Arrow does zero-copy with nested ~Structs + Schema. Have you already considered Apache Arrow?

https://arrow.apache.org/ :

## What is Arrow?
### Format
Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

Learn more about the design or read the specification.

### Libraries
Arrow's libraries implement the format and provide building blocks for a range of use cases, including high performance analytics. Many popular projects use Arrow to ship columnar data efficiently or as the basis for analytic engines.

Libraries are available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust. See how to install and

However, consider the C++ standard template library.  In Python,
everything is
an object, with a consistent handle.  Lists and tuples are uniform
sequences of
handles, dicts are uniform handle-to-handle mappings.  That's not so in
the C++
template library.  The whole point is to provide individual routines
to each variant of a data structure based on the template's type
variables.  It's
one-many, not one-one, from the point of view of your proposed FFI ABI. 
It seems
to me that more than Python's internal representation, which changes
fairly slowly
and is pretty well-documented, the target ABI is more variable, and if
the target
is as low-level as C there is not going to be one because equivalents of
structures will be defined per-project rather than for all C libraries.

> * This mechanism is transparent to users, there is modules in standard
> library
> to support it. [...] This mechanism provide user with maximum
> flexibility.

That sounds like ctypes to me.

> * almost zero-cost abstraction.

What does that mean?

> It just make some basic data representation conversion and
> invoke the method in dynamic library.

Still sounds like ctypes to me.

So I come back to the theme: what do you want that ctypes doesn't

Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IEHRGPWLUTB4OYB7OBIAF6OCPUO6S624/
Code of Conduct: http://python.org/psf/codeofconduct/