How about using modern C++ in development of CPython ?
Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI was a black box without any guarantees whatsoever. For any software that's going to dynamically link and exchange binary types with other independently produced software, that's a deal breaker. On 24.03.2021 12:54, redradist@gmail.com wrote:
Hi all,
What about of using modern C++ in developing CPython ?
With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ... _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SMF4B2SO... Code of Conduct: http://python.org/psf/codeofconduct/
-- Regards, Ivan
On Wed, 24 Mar 2021 19:45:49 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI was a black box without any guarantees whatsoever. For any software that's going to dynamically link and exchange binary types with other independently produced software, that's a deal breaker.
That depends if you use C++ internally or expose C++ bits in the public API. If you only use C++ internally, binary compatibility is presumably less of an issue. Regards Antoine.
On 24.03.2021 19:58, Antoine Pitrou wrote:
On Wed, 24 Mar 2021 19:45:49 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI was a black box without any guarantees whatsoever. For any software that's going to dynamically link and exchange binary types with other independently produced software, that's a deal breaker. That depends if you use C++ internally or expose C++ bits in the public API. If you only use C++ internally, binary compatibility is presumably less of an issue.
Python produces and accepts its internal types in API calls and allows extension modules to derive from them -- so it cannot "only use C++ internally" if those are going to become C++ types. (And if not, the point of using C++ is unclear.)
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GHYWBZY5... Code of Conduct: http://python.org/psf/codeofconduct/
-- Regards, Ivan
On Thu, 25 Mar 2021 20:22:55 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
On 24.03.2021 19:58, Antoine Pitrou wrote:
On Wed, 24 Mar 2021 19:45:49 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI was a black box without any guarantees whatsoever. For any software that's going to dynamically link and exchange binary types with other independently produced software, that's a deal breaker. That depends if you use C++ internally or expose C++ bits in the public API. If you only use C++ internally, binary compatibility is presumably less of an issue.
Python produces and accepts its internal types in API calls and allows extension modules to derive from them -- so it cannot "only use C++ internally" if those are going to become C++ types. (And if not, the point of using C++ is unclear.)
You can use C++ without exposing C++ types in the API. For example, C++ templates could improve the maintainability of the generic "stringlib" routines that are currently based on C macros. Another example is using RAII in function bodies to help cleanup owned references. Regards Antoine.
On 25/03/2021 18.39, Antoine Pitrou wrote:
On Thu, 25 Mar 2021 20:22:55 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
On 24.03.2021 19:58, Antoine Pitrou wrote:
On Wed, 24 Mar 2021 19:45:49 +0300 Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI was a black box without any guarantees whatsoever. For any software that's going to dynamically link and exchange binary types with other independently produced software, that's a deal breaker. That depends if you use C++ internally or expose C++ bits in the public API. If you only use C++ internally, binary compatibility is presumably less of an issue.
Python produces and accepts its internal types in API calls and allows extension modules to derive from them -- so it cannot "only use C++ internally" if those are going to become C++ types. (And if not, the point of using C++ is unclear.)
You can use C++ without exposing C++ types in the API. For example, C++ templates could improve the maintainability of the generic "stringlib" routines that are currently based on C macros. Another example is using RAII in function bodies to help cleanup owned references.
C has ways to handle cleanup and ownership better -- at least some compilers do. For example gcc has __attribute__(cleanup(cleanup_function)) for automatic cleanup on scope boundaries. If I recall correctly it works something like this: void Py_auto_decref(PyObject **o) { if (!o || !*o) return; Py_DECREF(*o); *o = NULL; } PyObject * func(PyObject self) { PyObject *spam __attribute__((cleanup(Py_auto_decref))); ... Py_RETURN_NONE; // spam gets automatically decrefed } It's too bad that the feature is not universally available in C99. Christian
@Christian Heimes RAII is more than cleaning resource at the end of function, it is that while you are using class - resource will be available ... You can move resource using modern C++ `std::move` ant lots of other features that could improve readability and maintainability of the code
24.03.21 11:54, redradist@gmail.com пише:
What about of using modern C++ in developing CPython ?
With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...
The drawbacks: 1. You can only use C++ internally. All API should be in pure C. It reduces possible benefits. 2. If you use C++, even internally, you have to link with libstdc++ and use C++ compatible linker. 3. C++ is much much more complex than pure C. It drastically reduces the number of available contributors and maintainers. Also note that the C code it already written. The C++ code has yet to be written, with chances to introduce new bugs during rewriting and distracting resources from other tasks.
The benefits: 1. You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. 2. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel. My personal stop of contributing in CPython is that it is written in pure C !! I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance
On Sat, Apr 17, 2021 at 3:20 AM <redradist@gmail.com> wrote:
The benefits:
1. You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. 2. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel.
Citations please?
My personal stop of contributing in CPython is that it is written in pure C !!
A lot of CPython is actually written in Python. There's plenty that you can contribute to without writing a single line of C code, if you so desire. ChrisA
Chris Angelico wrote:
On Sat, Apr 17, 2021 at 3:20 AM redradist@gmail.com wrote:
The benefits:
You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel.
Citations please?
What exactly do you what ?
On Sat, Apr 17, 2021 at 3:32 AM <redradist@gmail.com> wrote:
Chris Angelico wrote:
On Sat, Apr 17, 2021 at 3:20 AM redradist@gmail.com wrote:
The benefits:
You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel.
Citations please?
What exactly do you what ?
Evidence for your statements. Otherwise, empty statements can be rejected with empty rebuttals. ChrisA
Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU or read some articles ... otherwise I will need to spend too many time providing evidences to you and after all you will probably will reject anyway (because lots of people is biased and they even do not understand that, it is not about you, it is in general)
On 4/16/21 10:43 AM, redradist@gmail.com wrote:
Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU or read some articles ... otherwise I will need to spend too many time providing evidences to you and after all you will probably will reject anyway (because lots of people is biased and they even do not understand that, it is not about you, it is in general)
You are the one proposing the change, so it's up to you to provide the evidence for it. If you aren't willing to put in a few hours for that effort, why should we put the weeks and months to port the code over? -- ~Ethan~
Ethan Furman wrote:
On 4/16/21 10:43 AM, redradist@gmail.com wrote:
Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU or read some articles ... otherwise I will need to spend too many time providing evidences to you and after all you will probably will reject anyway (because lots of people is biased and they even do not understand that, it is not about you, it is in general) You are the one proposing the change, so it's up to you to provide the evidence for it. If you aren't willing to put in a few hours for that effort, why should we put the weeks and months to port the code over? -- ~Ethan~
I do not ask porting code, I ask using new code with C++ and if code was tested enough to reimplement it in C++ with RAII, <algorithms> Also I suggest using C++ excepting that most of the people here now it ... It was not intended to teach C++ here, especially in Mail List ))) And reason why you at least should try learn other languages, it is because it will make you better developer
On Fri, Apr 16, 2021 at 7:15 PM Denis Kotov <redradist@gmail.com> wrote:
On 4/16/21 10:43 AM, redradist@gmail.com wrote:
Take a look at this video https://www.youtube.com/watch?v=D7Sd8A6_fYU or read some articles ... otherwise I will need to spend too many time
You are the one proposing the change, so it's up to you to provide the evidence for it. If you aren't willing to put in a few hours for that effort, why should we put the weeks and months to
Ethan Furman wrote: providing evidences to you and after all you will probably will reject anyway (because lots of people is biased and they even do not understand that, it is not about you, it is in general) port the code over?
-- ~Ethan~
I do not ask porting code, I ask using new code with C++ and if code was tested enough to reimplement it in C++ with RAII, <algorithms>
Also I suggest using C++ excepting that most of the people here now it ... It was not intended to teach C++ here, especially in Mail List )))
And reason why you at least should try learn other languages, it is because it will make you better developer
Hi Denis, While I can accept that your intentions are honourable, did you stop to think that you are casting aspersions at a very capable and in many cases senior developers by suggesting that the reason they will not adopt C++ as an implementation language? You are correct that there is a deal of inertia behind C as the implementation language for CPython, as indeed there should be. It represents a huge investment, and has created valuable artefacts. As someone who isn't a core developer but manages programmers professionally it seems to me that you are ignoring many easily detectable issues, some technical and some social. - Who will put in the engineering effort to ensure that C++ code is supported within CPython codebase on all supported platforms? - Who will create and maintain the extra tests this would require? - Who will handle the inevitable deep bugs that the introduction of a not-fully-compatible technology will create? - By how much would such a change enlarge the core developer community? I so far know of one person it would add—you! What's the return on the effort? Remember, relatively few people are paid to work on CPython. Most do it for love and/or to scratch personal technical itches. What would they get out of the adoption of C++. While your enthusiasm is welcome, it's beginning to become a little wearing. Perhaps there's some history in the python-dev archives that would inform you of previous discussions and help you repeating already-considered arguments. I'm struggling to see the benefits here, and your presumption that experienced team members should immediately be persuaded by your arguments seems a little, well, presumptuous.
Code of Conduct: http://python.org/psf/codeofconduct/
Perhaps there's some history in the python-dev archives that would inform you of previous discussions and help you repeating already-considered arguments.
This topic has come up a few times over the years. Maybe it would be worthwhile to have an informational PEP which documents the various arguments pro and con to short-circuit or inform future discussions. I'm not volunteering to write it. Denis, maybe you could make a run at it. Skip
On Wed, Apr 21, 2021 at 1:05 AM Skip Montanaro <skip.montanaro@gmail.com> wrote:
Perhaps there's some history in the python-dev archives that would inform you of previous discussions and help you repeating already-considered arguments.
This topic has come up a few times over the years. Maybe it would be worthwhile to have an informational PEP which documents the various arguments pro and con to short-circuit or inform future discussions. I'm not volunteering to write it. Denis, maybe you could make a run at it.
The arguments should be being made the other way around. Status quo should not need to debate its own value; a proposed change needs to convince people of its worth. CPython is currently written in C, and anyone who suggests rewriting it has to be the one to submit evidence. (Or if not rewriting it, then whatever the change is. I'm really not sure what's being proposed - wholesale reimplementation? Permitting new modules to be written in C++? The ability to write extension modules in C++? In any case, it's been quite empty of supporting evidence.) ChrisA
On 4/16/21 10:27 AM, redradist@gmail.com wrote:
Chris Angelico wrote:
On Sat, Apr 17, 2021 at 3:20 AM redradist@gmail.com wrote:
The benefits:
You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel.
Citations please?
What exactly do you what ?
A list of current Python containers that are already in C++, that work exactly as our C ones do (otherwise, we will have had to add code to make them work as we want, and surely that will introduce bugs). Examples of how C++ is more maintainable. A poll of users that love C++ enough to contribute to CPython because we also use C++, combined with a poll of users currently contributing C code that would stop because they don't know/like C++. We should also have evidence that C++ is available on all supported platforms that we support CPython on. -- ~Ethan~
Hi, I used C++ in the past to write a small game. My experience was that the compilation was quite slow and many compiler errors were hard to understand. I love the fact the CPython is written in C because its build time is under one minute for a fresh build, and way faster for an incremental build (modifying a single C file). I'm part of the "trial-and-error" church. I modify random parts of the C code, build, test and repeat until it works as I want ;-) A *fresh* build (after make clean) of CPython on my laptop (8 threads, 4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh build in release mode (make -j10 and gcc -O3) takes 44 seconds on the same laptop. CPython is made of around 500K lines of C code. Victor On Fri, Apr 16, 2021 at 7:19 PM <redradist@gmail.com> wrote:
The benefits:
1. You will link with high quality libstdc++ with lots of reusable containers without writing your own "buggy" one. 2. C++ is much much more maintainable than pure C. It drastically increase number of contributors that what like writing high quality and maintainable code without reinventing the wheel.
My personal stop of contributing in CPython is that it is written in pure C !! I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YNVO3NM7... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
Guys, the issue is that I most of the time see that somebody used C++ for one or two times, did not understand it and left with bad taste ... Please, answer me question, if you will go in gym two times, will you get stop training and say that it does not fit in your life ?
Anyone who has done a language change on a project knows that it is a huge disruption. You need solid justification to make such a change. All I have seen in this thread is personal opinion. Since this is a personal opinion exchange, I am of the humble opinion that the personal opinions of core devs matter the most, since a language change would affect them more than anyone else. April 16, 2021 1:47 PM, redradist@gmail.com wrote:
Guys, the issue is that I most of the time see that somebody used C++ for one or two times, did not understand it and left with bad taste ...
Please, answer me question, if you will go in gym two times, will you get stop training and say that it does not fit in your life ? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWK... Code of Conduct: http://python.org/psf/codeofconduct
edwin＠211mainstreet.net wrote:
Anyone who has done a language change on a project knows that it is a huge disruption. You need solid justification to make such a change. All I have seen in this thread is personal opinion. Since this is a personal opinion exchange, I am of the humble opinion that the personal opinions of core devs matter the most, since a language change would affect them more than anyone else. April 16, 2021 1:47 PM, redradist@gmail.com wrote:
Guys, the issue is that I most of the time see that somebody used C++ for one or two times, did not understand it and left with bad taste ... Please, answer me question, if you will go in gym two times, will you get stop training and say that it does not fit in your life ? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWK... Code of Conduct: http://python.org/psf/codeofconduct
Okay lets try to discuss one by one: 1) Readability - less code, most code is hidden by abstraction without losing performance In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed with C++ std::shared_ptr<> (RustPython use analog Arc<>) 2) Maintainability - no code duplication in favor of using reusable classes In CPython I saw custom lists and stacks and so on ... with C++ it could be switched to std::list<>, std::stack<> and so on In CPython lots of custom implemented algorithms that could be changed by using C++ <algorithms> that support lots of types !! 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources Also reusable peaces of code will help of maintaining life-time of objects with RAII, not only at the end of function, but in general, because life-time resource will be bound to object life-time
April 16, 2021 2:08 PM, "Denis Kotov" <redradist@gmail.com> wrote:
edwin＠211mainstreet.net wrote:
Anyone who has done a language change on a project knows that it is a huge disruption. You need solid justification to make such a change. All I have seen in this thread is personal opinion. Since this is a personal opinion exchange, I am of the humble opinion that the personal opinions of core devs matter the most, since a language change would affect them more than anyone else. April 16, 2021 1:47 PM, redradist@gmail.com wrote: Guys, the issue is that I most of the time see that somebody used C++ for one or two times, did not understand it and left with bad taste ... Please, answer me question, if you will go in gym two times, will you get stop training and say that it does not fit in your life ? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LWJ4WGWK... Code of Conduct: http://python.org/psf/codeofconduct
Okay lets try to discuss one by one: 1) Readability - less code, most code is hidden by abstraction without losing performance In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed with C++ std::shared_ptr<> (RustPython use analog Arc<>)
So every single python extension library would have to be rewritten to handle all the new C++ apis? Sounds like an idea everyone will be very excited about.
On Fri, 16 Apr 2021 18:08:58 -0000 "Denis Kotov" <redradist@gmail.com> wrote:
Okay lets try to discuss one by one: 1) Readability - less code, most code is hidden by abstraction without losing performance In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed with C++ std::shared_ptr<> (RustPython use analog Arc<>)
std::shared_ptr<> would be a bad replacement for CPython's reference counting for two reasons: 1) the reference count is maintained in a separate memory block (unless you were careful enough to use std::make_shared() for allocation) 2) the reference count is atomic, and this has been proven to slow down CPython by 10-20%. That does not mean that CPython couldn't benefit from C++-based abstractions, but they would have to be implemented (or taken from another project such as pybind11). Regards Antoine.
Antoine Pitrou wrote: > On Fri, 16 Apr 2021 18:08:58 -0000 > "Denis Kotov" redradist@gmail.com wrote: > > Okay lets try to discuss one by one: > > > > Readability - less code, most code is hidden by abstraction without losing performance > > > > In CPython code lots of stuff like Py_INCREF, Py_DECREF .. it could be fixed with C++ std::shared_ptr<> (RustPython use analog Arc<>) > > std::shared_ptr<> would be a bad replacement for CPython's reference > counting for two reasons: > 1) the reference count is maintained in a separate memory block (unless > you were careful enough to use std::make_shared() for allocation) > 2) the reference count is atomic, and this has been proven to slow down > CPython by 10-20%. Rust have 2 ref count classes Rc and Arc. First is without atomic, single threaded, second is with atomic and could be used in multiple thread. It is possible to implement the same std::ref_ptr like Rc without atomic variables
On Sat, 17 Apr 2021, 3:51 am , <redradist@gmail.com> wrote:
Guys, the issue is that I most of the time see that somebody used C++ for one or two times, did not understand it and left with bad taste ...
I've got more than a decade and a half of experience with both C++ (dating back to the relatively low quality VC6 C++ runtime) and CPython (getting on towards 20 years now), and the value of adding a C++ runtime to the CPython runtime would be vastly smaller than the value of adding it to an arbitrary C project. If folks want rich data structures in the CPython implementation, we want them using the Python builtin types, not the C++ STL containers. If they want a type hierarchy, we want them using Python types, not C++ types. If they want automatic resource management, then we want them working out how to lift the affected code out of C and into Python (for example, the import system rewrite). In a lot of ways, CPython's C API can be viewed as one of the many competing approaches to enabling "C-with-objects" programming, just like C++. There are certainly nice features in modern C++ that make it easier to scale to large projects than vanilla C code, but the bulk of the CPython code base has never really been vanilla C code - it has always been able to rely on the builtin Python containers and type hierarchy for a lot of the heavy lifting. And it is already a bigger barrier to entry than we would like to ask potential contributors to the lower level code to learn two object models (the Python one and the much simpler C one), let alone if we were to start asking them to learn a 3rd one (C++) that has a deserved reputation as one of the most complex and error prone object models in widespread use. Cheers, Nick.
On Sun, 18 Apr 2021 02:13:57 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
If they want automatic resource management, then we want them working out how to lift the affected code out of C and into Python (for example, the import system rewrite).
There's a significant amount of wishful thinking here. When CPython code gets written in C, it's often because it's presumed to be performance-critical. The import system was indeed rewritten in Python... but some parts were then written back in C (see Python/import.c) to fix performance regressions.
In a lot of ways, CPython's C API can be viewed as one of the many competing approaches to enabling "C-with-objects" programming, just like C++.
It's a clumsy API to use *from C*. Witness the inconsistent behaviour of those APIs wrt. borrowed references, for example, or the fact that forgetting an INCREF or DECREF can lead to occasional leaks or crashes, or the delicate task of teaching the cyclic GC about a particular type, or all the special cases where those APIs deviate slightly in semantics from their pure Python equivalents, or the fact that APIs are added in adhoc manner, leading to an inconsistent set of primitives... (extracting a C long from a Python object works a bit differently from extracting a C unsigned long or long long... which then requires the caller to learn about those subtle differences and work around them). CPython's C API is probably fine for consumption by intermediate layers such as Cython. It's a maze to navigate for direct use. Regards Antoine.
On Sun, 18 Apr 2021, 2:47 am Antoine Pitrou, <antoine@python.org> wrote:
On Sun, 18 Apr 2021 02:13:57 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
If they want automatic resource management, then we want them working out
how
to lift the affected code out of C and into Python (for example, the import system rewrite).
There's a significant amount of wishful thinking here. When CPython code gets written in C, it's often because it's presumed to be performance-critical. The import system was indeed rewritten in Python... but some parts were then written back in C (see Python/import.c) to fix performance regressions.
Aye, I know, but the overall organisation of the import system still changed to "Python with C acceleration of key parts" rather than the previous "almost entirely C" approach. Other implementations are now free to reuse the Python parts, and only have to implement the accelerators separately.
In a lot of ways, CPython's C API can be viewed as one of the many competing approaches to enabling "C-with-objects" programming, just like C++.
It's a clumsy API to use *from C*. Witness the inconsistent behaviour of those APIs wrt. borrowed references, for example, or the fact that forgetting an INCREF or DECREF can lead to occasional leaks or crashes, or the delicate task of teaching the cyclic GC about a particular type, or all the special cases where those APIs deviate slightly in semantics from their pure Python equivalents, or the fact that APIs are added in adhoc manner, leading to an inconsistent set of primitives...
Right, but that clumsiness isn't any easier to manage from C++ than it is from C. Things like RAII rely on consistent behaviour from the resources being managed, and the current C API doesn't offer that. For example, putting a borrowed or stolen reference into an RAII based reference manager will result in an early free or a resource leak, just as happens when you decref a borrowed reference or incref a stolen one in C. Hence the need for wrapper APIs and tools that make the C API more C++ friendly rather than it being straightforward to use directly. CPython's C API is probably fine for consumption by intermediate layers
such as Cython. It's a maze to navigate for direct use.
That I also agree with, and hence I'd definitely be a fan if we ever figured out a way to bootstrap Cython into the CPython build process so that accelerated standard library modules could be written in Cython. Unlike C++, Cython uses the same data structures and object model as Python does, and it already smooths over the irregularities in the C API to allow for fully automated resource management. The Python based syntax can even help lower the barriers to entry for folks that don't already know C (although you do still need to learn the underlying C memory model). Cheers, Nick.
On 16/04/2021 19.39, Victor Stinner wrote:
A *fresh* build (after make clean) of CPython on my laptop (8 threads, 4 CPU cores) takes 13 seconds using make -j10 and gcc -O0. A fresh build in release mode (make -j10 and gcc -O3) takes 44 seconds on the same laptop.
$ make clean $ time make -j10 ... real 0m2,072s user 0m4,715s sys 0m2,333s ./configure -C and ccache are fantastic. Christian
On 16/04/2021 19.14, redradist@gmail.com wrote:
My personal stop of contributing in CPython is that it is written in pure C !! I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance
There are plenty of Open Source projects that could use more capable C++ developers. :) I'm not a fan of C++. It has its use cases, e.g. in UI. Python core isn't the best fit. AFAIK most core devs are not fluent in C++. Despite it's name, C++ is really a different language than C. It has a different ABI and stdlib than C, too. In my personal opinion C++ won't give us any net benefits. I'd much rather go for Rust than C++ to gain memory safety. Christian
Christian Heimes wrote:
On 16/04/2021 19.14, redradist@gmail.com wrote:
My personal stop of contributing in CPython is that it is written in pure C !! I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance There are plenty of Open Source projects that could use more capable C++ developers. :) I'm not a fan of C++. It has its use cases, e.g. in UI. Python core isn't the best fit. AFAIK most core devs are not fluent in C++. Despite it's name, C++ is really a different language than C. It has a different ABI and stdlib than C, too. In my personal opinion C++ won't give us any net benefits. I'd much rather go for Rust than C++ to gain memory safety. Christian
Rust is not syntax compatible with C and that is why I suggest to use C++. If you are not fan of C++, with Rust there are also templates and sometimes even harder errors for life-time than C++ error template instantiation ))) Rust does not fit in 30 years code base written in C like C++ does
Denis, There's a standard way of interesting the Python community in new syntax (and C++, while not Python, is new syntax in spades to this community ;-). You take a hunk of the standard library (in this case it would have to be an accelerator written in C since you want to compare C++ vs. C) or interpreter code, and translate it to the new syntax. Now, *INCREF and friends are frequently cited as annoyances or even warts, so your suggestion of std::shared _ptr<> seemed plausible to me. But Antoine peeked at it and points out it's not that easy, for performance reasons you can't use it "as is". It's true that you could reimplement it, but then it's not std::shared_ptr<> any more and the only benefit left is that it looks familiar to C++ programmers (and may mislead *them* into thinking about it as if it were exactly std::shared_ptr<>). And that's why you need to do more work than arguing that in principle C++ is just a better language than C. We've been hearing that for 4 decades now (at least we greybeards have), and we've discovered that for many existing applications, C++ may be better but the cost of converting large swaths of C code to equivalent C++ that passes all tests is too big. Python may very well be one of them. So if you're not going to do the work to demonstrate big wins from using C++ instead of C in actual Python implementation code, I think you're wasting your posts. Steve
Stephen J. Turnbull wrote:
You take a hunk of the standard library (in this case it would have to be an accelerator written in C since you want to compare C++ vs. C) or interpreter code, and translate it to the new syntax. Now, *INCREF and friends are frequently cited as annoyances or even warts, so your suggestion of std::shared _ptr<> seemed plausible to me. But Antoine peeked at it and points out it's not that easy, for performance reasons you can't use it "as is". It's true that you could reimplement it, but then it's not std::shared_ptr<> any more and the only benefit left is that it looks familiar to C++ programmers (and may mislead *them* into thinking about it as if it were exactly std::shared_ptr<>).
Re-implementing of std::shared _ptr<> for single thread is smallest of the problems
And that's why you need to do more work than arguing that in principle C++ is just a better language than C. We've been hearing that for 4 decades now (at least we greybeards have), and we've discovered that for many existing applications, C++ may be better but the cost of converting large swaths of C code to equivalent C++ that passes all tests is too big. Python may very well be one of them. So if you're not going to do the work to demonstrate big wins from using C++ instead of C in actual Python implementation code, I think you're wasting your posts.
I thought about it, but will CPython accept the PR for this changes if it would show benefits ?
On Mon, 17 Jan 2022 at 06:52, Denis Kotov <redradist@gmail.com> wrote:
And that's why you need to do more work than arguing that in principle C++ is just a better language than C. We've been hearing that for 4 decades now (at least we greybeards have), and we've discovered that for many existing applications, C++ may be better but the cost of converting large swaths of C code to equivalent C++ that passes all tests is too big. Python may very well be one of them. So if you're not going to do the work to demonstrate big wins from using C++ instead of C in actual Python implementation code, I think you're wasting your posts.
I thought about it, but will CPython accept the PR for this changes if it would show benefits ?
We can't say that, no. The point here is that if you *don't* write such a PR and demonstration, nothing will change. If you do, then *maybe* it will get accepted, it depends if the benefits demonstrated by the PR are sufficient. The question is whether just having a *chance* of getting it in is sufficient to persuade you to produce such a PR. Paul.
On Fri, Apr 16, 2021 at 11:13 AM Christian Heimes <christian@python.org> wrote:
My personal stop of contributing in CPython is that it is written in
On 16/04/2021 19.14, redradist@gmail.com wrote: pure C !!
I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance
There are plenty of Open Source projects that could use more capable C++ developers. :)
I'm not a fan of C++. It has its use cases, e.g. in UI. Python core isn't the best fit. AFAIK most core devs are not fluent in C++. Despite it's name, C++ is really a different language than C. It has a different ABI and stdlib than C, too. In my personal opinion C++ won't give us any net benefits. I'd much rather go for Rust than C++ to gain memory safety.
Agreed. Rust would be much better than C++. Rust actually brings something new and interesting to the discussion. C++ is "almost Java", but not as good. The Linux kernel tried C++, but backed away from it. But now it's adding Rust. I've had the experience, in the past, of writing code in C++ only to find that its intended users refused to use it /because/ it was in C++. In retrospect, I'm not entirely disappointed that they did - it steered me toward sticking with bash and C, and getting more into Python.
I heard that rust has fewer target architectures. Is that a minus point with regards to C? Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-RahmaanJ> Mauritius
On Thu, Jan 20, 2022 at 8:16 PM Dan Stromberg <drsalists@gmail.com> wrote:
On Fri, Apr 16, 2021 at 11:13 AM Christian Heimes <christian@python.org> wrote:
My personal stop of contributing in CPython is that it is written in
On 16/04/2021 19.14, redradist@gmail.com wrote: pure C !!
I wrote code in both: pure C and C++, but I like writing code in C++, because it simplifies things without losing perfomance
There are plenty of Open Source projects that could use more capable C++ developers. :)
I'm not a fan of C++. It has its use cases, e.g. in UI. Python core isn't the best fit. AFAIK most core devs are not fluent in C++. Despite it's name, C++ is really a different language than C. It has a different ABI and stdlib than C, too. In my personal opinion C++ won't give us any net benefits. I'd much rather go for Rust than C++ to gain memory safety.
Agreed.
Rust would be much better than C++.
At least one Python runtime has been written in Rust already: https://github.com/RustPython/RustPython The fundamental reason we haven't embarked on new language things in CPython is compatibility and complexity. If there were clear demonstrable wins to adopting a language in addition to C for the CPython runtime, we could consider it (ultimately as a PEP with reasoning, alternatives, and rejected ideas all laid out). But it isn't going to be done on a whim as it a new language doesn't magically solve problems, it just adds more problems. Of the three things listed at the start of this thread, the first two don't exist. From huge codebase experience with C++, it does not cause significantly better (1) Readabillity or (2) Maintainability on its own compared to C - both of those are entirely up to the software engineering practices adopted within the project no matter what the language is. (3) RAII is the number one thing I would enjoy having in the CPython internals. Manually managing Py_DECREFs is inhumane. As a result we have built up infrastructure to detect leaks in our test suite so new refcount leak problems *usually* don't happen - so long as test coverage is in place (ex: 3.10.2...). And RAII isn't magic, even it can be used wrong. It's just that the location and spelling of the wrong changes. Rewrites might sound "easy" (emphasis on the air quotes! nobody should think it's easy)... but only until you actually have to match API and ABI and bug for bug compatibility with the entire corpus of existing real world code. Just ask PyPy. ;) -gps
From huge codebase experience with C++, it does not cause significantly better (1) Readabillity or (2) Maintainability on its own compared to C
I would argue with you on that ... RAII is fundamental C++ feature that improves Maintainability !! Also readability is much better because you work with object that will be compiled in the same code if you will write it by hands, for example: std::vector
Denis Kotov writes:
From huge codebase experience with C++, it does not cause significantly better (1) Readabillity or (2) Maintainability on its own compared to C
But that's not what we're talking about. "Port CPython to C++" is a perennial suggestion that gets rejected fairly quickly, as the C++ stdlib equivalent structures and functions in CPython are efficient and well-tested. I don't know much about Rust but I know that some core modules have already been ported to Rust. Since core devs, with a specific reason (security, it's crypto stuff) to move away from C are involved in that effort I would guess that movint to Rust is far more likely than to C++. The point of C++ standard support level is linking CPython to external codebases using C++, and at least for the standard CPython currently supports, the C++ ABI is specific to each compiler and version (for some compilers), right?
Stephen J. Turnbull wrote:
The point of C++ standard support level is linking CPython to external codebases using C++, and at least for the standard CPython currently supports, the C++ ABI is specific to each compiler and version (for some compilers), right?
Yes, C++ ABI is specific to each compiler, but it is not a problem, because you will anyway recompile CPython for each compiler !! Also g++ and clang++ has the same C++ ABI !! Usually people say that C++ not fit in CPython not knowing C++, it is common thing in psychology: https://en.wikipedia.org/wiki/I_know_that_I_know_nothing https://medium.com/@cogitality/the-dunning-kruger-effect-a-paradox-that-does... Lets consider the advantages of C++: 1) C++ has objects, Python has objects, C do not have objects (struct is not object in term of OOP) 2) C++ has templates that allow to generate for free function specialization to improve performance 3) C++ has move semantic and switching to C++ you have move object for free !! 4) C++ has RAII (equivalent to CPython __del__), that works the same way to 5) ... and so on ... ... but i will not convince you because it is also another psychology stuff, if you what to believe in something, you will try to find evidences for proving your theory, see also: https://www.youtube.com/watch?v=vUzGV4F7He8 Please, to not take it personally, i just shared my thought about all of this discussion ... I just optimistic all my life that is why I try to impact, even if it is almost impossible )))
Denis Kotov writes:
Yes, C++ ABI is specific to each compiler, but it is not a problem, because you will anyway recompile CPython for each compiler !!
Right, but the point is that we want few C++ compilers people really use to get upset by Python code. Since most changes are backward compatible (or new compilers may have a backward compatibility option) specifying an older standard accomplishes that goal.
Usually people say that C++ not fit in CPython not knowing C++,
That's not the issue. There are plenty of people who know C++ who say the benefits of C++ are not worth the bugs, the effort, and the plain old churn that a rewrite in C++ (even if incremental) would require.
Stephen J. Turnbull wrote:
Usually people say that C++ not fit in CPython not knowing C++, That's not the issue. There are plenty of people who know C++ who say
Denis Kotov writes: the benefits of C++ are not worth the bugs, the effort, and the plain old churn that a rewrite in C++ (even if incremental) would require.
I am not sure about this one ... For example, PyObject is much better to implement using C++ class, it will bring RAII that will reduce number of issues not free memory in CPython See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o Without RAII - introduced the most dangerous errors !!
Denis Kotov writes:
For example, PyObject is much better to implement using C++ class, it will bring RAII that will reduce number of issues not free memory in CPython
Well, yes, that's the C++ Kool-Aid we're all supposed to drink. But it has its costs, too. How does it affect performance? How does it interact with slots, __new__, metaclasses, and other features of the Python class hierarchy, especially when coded as C? What about translating all of the accelerated modules written in C using the existing stable ABI? Have you looked at how this would affect NumPy, Pandas, and cffi? That's just a bunch of random related software, there are plenty of others (not to mention non-public software that neither of us has access to that may be affected). Really, you need to bring new information specific to Python rather than this generic "C++ is great" advocacy.
See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o
No thanks, if it's not text I'm not going to look at it. I don't have time to watch videos with no explanation, and best guess is 90% of it I already know, just as I don't think you've yet written anything I didn't already know.
Without RAII - introduced the most dangerous errors !!
List the CVEs. I'll happily go read those! Steve
Stephen J. Turnbull wrote:
Denis Kotov writes:
See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o No thanks, if it's not text I'm not going to look at it. I don't have time to watch videos with no explanation, and best guess is 90% of it I already know, just as I don't think you've yet written anything I didn't already know.
It is just 10-20 seconds, I used YouTube Shorts to slice it :)
On Mon, 17 Oct 2022 at 11:20, Denis Kotov <redradist@gmail.com> wrote:
Stephen J. Turnbull wrote:
Denis Kotov writes:
See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o No thanks, if it's not text I'm not going to look at it. I don't have time to watch videos with no explanation, and best guess is 90% of it I already know, just as I don't think you've yet written anything I didn't already know.
It is just 10-20 seconds, I used YouTube Shorts to slice it :)
So while I agree with Stephen that if someone expects me to watch a video I almost certainly won't (no matter how short) I watched this, just to make the point. And indeed, it says nothing that I (and I imagine Stephen) didn't know, and all it does is list a bunch of vulnerabilities and assert that C++ can help with these. And it's presented by Herb Sutter, so yes, I'm sure he knows what he's talking about. And indeed, I know C++ so I can confirm that C++ has idioms and language features that you can use to help with mitigating these vulnerabilities. But it's not a magic wand, you still have to write the code to use these features. You can write pure C code in C++, so there's no instant fix here, you have to **change the code** to gain the benefits you suggest. If you don't have a *specific* explanation of how Python can be improved by rewriting it in C++, that takes into account all of the costs of such a rewrite, then you won't get anywhere here. I could just as easily say "yay, Rust is great, let's use Rust". Or if you're that convinced, do the work. Make a fork of Python that uses C++ instead of C, demonstrate that it's (in whatever way you want to measure) "better" than the existing implementation, and be prepared to defend that position. That's what multiple people have done, when suggesting ways to speed up Python, ways to remove or mitigate the GIL, etc. That route *works*, if your argument is sound. But just saying "you should do X" and expecting people to just do the work for you is almost certainly *not* going to work. Paul
CPython is written (mostly) in C. There's Jython written in Java. There also IronPython written on C#. Feel free to write your own Python implementation in C++. Could be fun!
On 17/10/22 10:13 pm, Denis Kotov wrote:
For example, PyObject is much better to implement using C++ class,
I used to think this. Python has OO features, C++ has OO features, should be a good match, right? Well, no. Python's OO is very flexible and dynamic, C++'s is very rigid and static. CPython uses various low-level tricks that rely on structs being just "plain old data". The OO features of C++ wouldn't help nuch with that at all, and would often get in the way. If you want a Python implementation that's less prone to memory management errors, I would suggest having a small kernel written in C, and write the rest using a tool similar to Cython that will take care of all the tricky reference counting for you. -- Greg
Haven't we migrated to Discourse? This discussion will probably not have any effects on this mailing-list. (yes, not everyone likes Discourse, and I'm skeptical as well, but the decision has been made by now) On Mon, 17 Oct 2022 09:13:34 -0000 "Denis Kotov" <redradist@gmail.com> wrote:
Stephen J. Turnbull wrote:
Usually people say that C++ not fit in CPython not knowing C++, That's not the issue. There are plenty of people who know C++ who say
Denis Kotov writes: the benefits of C++ are not worth the bugs, the effort, and the plain old churn that a rewrite in C++ (even if incremental) would require.
I am not sure about this one ... For example, PyObject is much better to implement using C++ class, it will bring RAII that will reduce number of issues not free memory in CPython See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o
Without RAII - introduced the most dangerous errors !!
People are free to continue to use python-dev, it just no longer has any official status, and many core devs and other key people have probably unsubscribed. I certainly have ignored this discussion here, even though (our of nostalgia, mostly) I am still subscribed. --Guido On Wed, Oct 19, 2022 at 9:11 AM Antoine Pitrou <antoine@python.org> wrote:
Haven't we migrated to Discourse? This discussion will probably not have any effects on this mailing-list.
(yes, not everyone likes Discourse, and I'm skeptical as well, but the decision has been made by now)
On Mon, 17 Oct 2022 09:13:34 -0000 "Denis Kotov" <redradist@gmail.com> wrote:
Stephen J. Turnbull wrote:
Usually people say that C++ not fit in CPython not knowing C++, That's not the issue. There are plenty of people who know C++ who say
Denis Kotov writes: the benefits of C++ are not worth the bugs, the effort, and the plain old churn that a rewrite in C++ (even if incremental) would require.
I am not sure about this one ... For example, PyObject is much better to implement using C++ class, it will bring RAII that will reduce number of issues not free memory in CPython See: https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o
Without RAII - introduced the most dangerous errors !!
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3JYIICCL... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
How does C++ fare in binary compatibility? Last time I checked it out (about 10 years ago), there was completely none, every compiler's ABI https://www.credosystemz.com/courses/aws-training-chennai/
Yes, each compiler implement its own compiler ABI, but for me it is not a problem at all, just provide build with 3 major compilers gcc/clang (due to binary compatability) and Visual Studio
participants (23)
-
Abdur-Rahmaan Janhangeer -
Antoine Pitrou -
Chris Angelico -
Christian Heimes -
Dan Stromberg -
Denis Kotov -
edwin＠211mainstreet.net -
Ethan Furman -
gopinathinchennai01＠gmail.com -
Greg Ewing -
Gregory P. Smith -
Guido van Rossum -
Ivan Pozdeev -
Matthias Görgens -
Nick Coghlan -
Paul Moore -
redradist＠gmail.com
-
Serhiy Storchaka -
Skip Montanaro -
Stephen J. Turnbull -
Stephen J. Turnbull -
Steve Holden -
Victor Stinner