{Python-ideas] C-API exposure
As I have been toying around with a few things, I have noticed that the C-API provides a lot more functionality than is exposed in Python. Much of the functionality can be reproduced one way or another. However, I was wondering if it would be feasible (and tractable) to expose every bit of the C-API in python.
If it happened, then everything in there could be written in pure python relative to all the other exposed pieces. This would allow easier prototyping of new language features. It would not be practical from a performance standpoint for most stuff, but it would help people understand how python works underneath. As well, exposing all the pieces would provide a way to test the C-API completely from pure python.
While I see several good things, I also see the size of the task. Just exposing the C-API would be a feat. On top of that, emulating the innards of each piece in pure python using the other exposed pieces would be a big job. Would it be worth it? Would it expose things we actually don't want exposed?
I think it would be really cool, but half the time that is a good warning sign.
-eric
On Tue, Mar 29, 2011 at 10:47 AM, Eric Snow ericsnowcurrently@gmail.com wrote:
As I have been toying around with a few things, I have noticed that the C-API provides a lot more functionality than is exposed in Python. Much of the functionality can be reproduced one way or another. However, I was wondering if it would be feasible (and tractable) to expose every bit of the C-API in python.
If it happened, then everything in there could be written in pure python relative to all the other exposed pieces. This would allow easier prototyping of new language features. It would not be practical from a performance standpoint for most stuff, but it would help people understand how python works underneath. As well, exposing all the pieces would provide a way to test the C-API completely from pure python.
While I see several good things, I also see the size of the task. Just exposing the C-API would be a feat. On top of that, emulating the innards of each piece in pure python using the other exposed pieces would be a big job. Would it be worth it? Would it expose things we actually don't want exposed?
I think it would be really cool, but half the time that is a good warning sign.
Well, would it really be pure Python? You should carefully consider how portable that "pure Python" code you propose to write would be to alternate Python implementations like Jython, IronPython or PyPy.
It also sounds like you're about to independently discover Cython.
Finally, can you be specific? Do you have some examples of C-APIs that could be exposed? What would be gained?
Certainly neither the new builtins nor the "pure" Python extrapolations would be portable. And I wasn't suggesting that they be exposed in the builtins module, but rather in their own module. By nature they would be implementation specific. However, they would be as insightful as poking around the C-API is (which I found to be very), but in Python.
As far as Cython goes, I am not terribly familiar with it. However, I think it is a sort of opposite. Cython seems to push Python down into C. The C-API builtins would push the C into Python (that doesn't sound good).
Regardless, I think doing this would take too much work to be worth it. But I did want to get the idea out there. I starting thinking about this when I was messing around with exec_closure. While it has proven superfluous, working on it exposed me to all the pieces in the C-API that do not have counterparts in Python. Things like cell objects. There are things in there that you can emulate, but not in an explicit way (like PyEval_EvalCodeEx).
It seems like as time has gone by, more of the internals have been exposed, like the AST module, the types module, metaclasses, the dis module, and others. Certainly these are not run-of-the-mill modules, and neither would this be. Those others have come about as needs have presented. I expect that will continue to be the case. The idea here was to skip to the chase and just expose the whole API.
One of my key questions is, what are the dangers in doing so? Security? Risk of fostering hacks? More people relying on implementation specific details? Enabling code that is incongrous with the Python vision? These are questions to which I am trying to find answers as I dive into the python-dev world. I appreciate the feedback by the way!
-eric
On Tue, Mar 29, 2011 at 11:53 AM, Guido van Rossum guido@python.org wrote:
On Tue, Mar 29, 2011 at 10:47 AM, Eric Snow ericsnowcurrently@gmail.com wrote:
As I have been toying around with a few things, I have noticed that the C-API provides a lot more functionality than is exposed in Python. Much
of
the functionality can be reproduced one way or another. However, I was wondering if it would be feasible (and tractable) to expose every bit of
the
C-API in python.
If it happened, then everything in there could be written in pure python relative to all the other exposed pieces. This would allow easier prototyping of new language features. It would not be practical from a performance standpoint for most stuff, but it would help people
understand
how python works underneath. As well, exposing all the pieces would
provide
a way to test the C-API completely from pure python.
While I see several good things, I also see the size of the task. Just exposing the C-API would be a feat. On top of that, emulating the
innards
of each piece in pure python using the other exposed pieces would be a
big
job. Would it be worth it? Would it expose things we actually don't
want
exposed?
I think it would be really cool, but half the time that is a good warning sign.
Well, would it really be pure Python? You should carefully consider how portable that "pure Python" code you propose to write would be to alternate Python implementations like Jython, IronPython or PyPy.
It also sounds like you're about to independently discover Cython.
Finally, can you be specific? Do you have some examples of C-APIs that could be exposed? What would be gained?
-- --Guido van Rossum (python.org/~guido)
On Tue, Mar 29, 2011 at 11:30 AM, Eric Snow ericsnowcurrently@gmail.com wrote:
Certainly neither the new builtins nor the "pure" Python extrapolations would be portable. And I wasn't suggesting that they be exposed in the builtins module, but rather in their own module. By nature they would be implementation specific. However, they would be as insightful as poking around the C-API is (which I found to be very), but in Python. As far as Cython goes, I am not terribly familiar with it. However, I think it is a sort of opposite. Cython seems to push Python down into C. The C-API builtins would push the C into Python (that doesn't sound good). Regardless, I think doing this would take too much work to be worth it. But I did want to get the idea out there. I starting thinking about this when I was messing around with exec_closure. While it has proven superfluous, working on it exposed me to all the pieces in the C-API that do not have counterparts in Python. Things like cell objects. There are things in there that you can emulate, but not in an explicit way (like PyEval_EvalCodeEx). It seems like as time has gone by, more of the internals have been exposed, like the AST module, the types module, metaclasses, the dis module, and others. Certainly these are not run-of-the-mill modules, and neither would this be. Those others have come about as needs have presented. I expect that will continue to be the case. The idea here was to skip to the chase and just expose the whole API.
Well people already do this using ctypes...
One of my key questions is, what are the dangers in doing so? Security? Risk of fostering hacks? More people relying on implementation specific details? Enabling code that is incongrous with the Python vision? These are questions to which I am trying to find answers as I dive into the python-dev world. I appreciate the feedback by the way!
Any or all of the above, probably, depending on the specific API you're considering...
You seem to have ignored my suggestion to think about how this would work in other Python interpreters.
Also many of the C APIs have subtle reference count behavior -- you don't want to have to worry about refcounting bugs *in your Python code*.
On Tue, Mar 29, 2011 at 12:47 PM, Guido van Rossum guido@python.org wrote:
On Tue, Mar 29, 2011 at 11:30 AM, Eric Snow ericsnowcurrently@gmail.com wrote:
Certainly neither the new builtins nor the "pure" Python extrapolations would be portable. And I wasn't suggesting that they be exposed in the builtins module, but rather in their own module. By nature they would be implementation specific. However, they would be as insightful as poking around the C-API is (which I found to be very), but in Python. As far as Cython goes, I am not terribly familiar with it. However, I
think
it is a sort of opposite. Cython seems to push Python down into C. The C-API builtins would push the C into Python (that doesn't sound good). Regardless, I think doing this would take too much work to be worth it.
But
I did want to get the idea out there. I starting thinking about this
when I
was messing around with exec_closure. While it has proven superfluous, working on it exposed me to all the pieces in the C-API that do not have counterparts in Python. Things like cell objects. There are things in there that you can emulate, but not in an explicit way (like PyEval_EvalCodeEx). It seems like as time has gone by, more of the internals have been
exposed,
like the AST module, the types module, metaclasses, the dis module, and others. Certainly these are not run-of-the-mill modules, and neither
would
this be. Those others have come about as needs have presented. I expect that will continue to be the case. The idea here was to skip to the
chase
and just expose the whole API.
Well people already do this using ctypes...
Haven't used them myself yet. So you can use them to expose all of the
C-API? Cool!
One of my key questions is, what are the dangers in doing so? Security? Risk of fostering hacks? More people relying on implementation specific details? Enabling code that is incongrous with the Python vision? These are questions to which I am trying to find answers as I dive into the python-dev world. I appreciate the feedback by the way!
Any or all of the above, probably, depending on the specific API you're considering...
I figured as much. :( Unless I have missed a doc somewhere, it seems like
it is a process of time to get a feel for what belongs in Python and what doesn't.
You seem to have ignored my suggestion to think about how this would work in other Python interpreters.
I really am not sure. It seems like there are already several modules in
the stdlib that are implementation specific, like dis. This would fall in that category. But maybe we are trying to lock or eliminate that category?
Also many of the C APIs have subtle reference count behavior -- you don't want to have to worry about refcounting bugs *in your Python code*.
That is one that I definitely hadn't thought about. I am totally with you
on that. Chalk that up to one more obstacle, though I suppose you would have to deal with it with a ctypes approach as well.
--
--Guido van Rossum (python.org/~guido)
I don't want this discussion to be an abuse of people's time to the benefit of my understanding, but I am finding these threads to be very insightful. So, thanks!
-eric
Den 29.03.2011 21:15, skrev Eric Snow:
Well people already do this using ctypes...Haven't used them myself yet. So you can use them to expose all of the C-API? Cool!
Yes, ctypes.pythonapi exposes the Python C API to Python.
You can use it for evil code like this hack to prevent thread switch (doesn't work with Python 3, as the implementation has changed). Just make sure you don't call extension code that releases the GIL, or all bets are off ;-)
Sturla
from contextlib import contextmanager import ctypes _Py_Ticker = ctypes.c_int.in_dll(ctypes.pythonapi,"_Py_Ticker")
@contextmanager def atomic(): tmp = _Py_Ticker.value _Py_Ticker.value = 0x7fffffff yield _Py_Ticker.value = tmp - 1
Now we can do
with atomic(): # whatever pass
On Tue, Mar 29, 2011 at 12:32 PM, Sturla Molden sturla@molden.no wrote:
Den 29.03.2011 21:15, skrev Eric Snow:
Well people already do this using ctypes...
Haven't used them myself yet. So you can use them to expose all of the C-API? Cool!
Yes, ctypes.pythonapi exposes the Python C API to Python.
You can use it for evil code like this hack to prevent thread switch (doesn't work with Python 3, as the implementation has changed). Just make sure you don't call extension code that releases the GIL, or all bets are off ;-)
Sturla
from contextlib import contextmanager import ctypes _Py_Ticker = ctypes.c_int.in_dll(ctypes.pythonapi,"_Py_Ticker")
@contextmanager def atomic(): tmp = _Py_Ticker.value _Py_Ticker.value = 0x7fffffff yield _Py_Ticker.value = tmp - 1
Now we can do
with atomic(): # whatever pass
Huh. That's terrifying. Thanks for the example.
Geremy Condra
On Wed, Mar 30, 2011 at 5:32 AM, Sturla Molden sturla@molden.no wrote:
from contextlib import contextmanager import ctypes _Py_Ticker = ctypes.c_int.in_dll(ctypes.pythonapi,"_Py_Ticker")
@contextmanager def atomic(): tmp = _Py_Ticker.value _Py_Ticker.value = 0x7fffffff yield _Py_Ticker.value = tmp - 1
Yikes, at least stick a try-finally in there!
If you must practice evil, practice safe evil ;)
Cheers, Nick.
Den 30.03.2011 06:00, skrev Nick Coghlan:
Yikes, at least stick a try-finally in there! If you must practice evil, practice safe evil ;)
I wasn't suggestion that one should actually do this. It was jut to show that the C API is exposed to Python.
Well, _Py_Ticker is not even in the C API, but it's not declared static so we can do bad things with it from outside.
The point is still that ctypes.pythonapi is the DLL containing the CPython interpreter.
Sturla
On Mar 29, 2011, at 12:15 PM, Eric Snow wrote:
I don't want this discussion to be an abuse of people's time to the benefit of my understanding, but I am finding these threads to be very insightful. So, thanks!
The discussion has made for an interesting read, so I don't think it has been a waste of time. The python-ideas mailing list is a reasonable place for flights of fancy and random musings :-)
That being said, python-ideas would be a little more sane (and less disconcerting) if the musings came in the form of "here's my wild idea, let's play with it" rather than "i don't fully understand the language we've got but am going to propose changing it anyway." If someone proposes to demolish 20 years worth of language success, it's harder to respond with an open-mind and a playful out-of-the-box outlook.
Raymond
participants (6)
-
Eric Snow -
geremy condra -
Guido van Rossum -
Nick Coghlan -
Raymond Hettinger -
Sturla Molden