Pymalloc and backward compatibility
I've been watching the pymalloc discussions going on recently, and I'm concerned that there may be some fairly major compatibility changes coming. I'd like to check that I understand the intention correctly. To provide some background - I wrote the Python interface for the Vim editor, a long time ago. At the time, Python 1.4 was the current version. I was *very* new to Python at the time, and this was my first real C extension (and is still my biggest - I haven't had much of a need to write my own C extensions since...) I didn't particularly understand the details of the API, so I basically just copied code from existing samples, and was pleased at how easily it all worked. The code has survived essentially unchanged through to Python 2.2, and still works fine. However, I'd be surprised if it doesn't take some fairly extreme liberties (mixing API "families", relying on all Del/Free calls mapping to the same underlying free(), etc). I can't even risk "fixing" it as I don't have any way of checking which API calls appeared in which version (I recall a message in one of the pymalloc threads to the effect that 1.4-compatible code was bound to include some of these bugs, because certain APIs weren't available in 1.4) OK, so maybe it's time to give up on Python 1.4 compatibility. But ideally, I'd prefer to go for an approach which leaves the code used under Python 2.2 and earlier completely unchanged, and only adds the minimum necessary for newer versions. Longer term, it might be worth considering upgrading the code to use some of the newer features in the Python API, but in all honesty that's not a great priority for me (and unlikely to be even remotely on the cards until I can realistically desupport everything before 2.0...) It seems to me that there are two options open for me: 1. The Python API from 1.4 is still guaranteed to work unchanged (even given the liberties mentioned above). In that case, do nothing. 2. There are changes required. In that case, make them, protected by python API checks (annoying ifdef fun). While (1) is superficially nicer from my point of view, it just defers the problem - at *some* point, the backward compatibility load becomes too much. So the 1.4-compatible code is deprecated, at best. In that case, I'd rather change things now, while the issues are fresh. The problem with (2) is that my immediate reaction is to do something like (pseudocode because I'm too lazy to look up the real API names and calls) #ifdef PYTHON_VERSION >= 2.3 PyObject *myObj = Py_WhizzyNewAllocator(); #else PyObject *myObj = Py_CruftyOldAllocator(); #endif That works fine - and looks OK from a maintenance standpoint, as long as the new API really is new. If the new API merely blesses existing calls as the "official right thing", the above code is going to look odd to a future maintainer (Why is the call to Py_WhizzyNewAllocator only enabled for 2.3 onwards? That API call was available from 1.5.2...) So I think what I'm saying is that my preference is for a completely new set of allocation APIs for Python 2.3, with all old APIs deprecated (and scheduled for removal at some time in the future). This should be clearly documented in the manual, including a list of all deprecated APIs to allow extension writers to grep for code that needs changing. Sample ifdef-protected type code like the above, for people like me who tend to treat all this as black magic anyway, would probably be a nice bonus. If this is what was planned anyway, just ignore me. I'll go back to sleep now....... Paul.
Moore, Paul wrote:
I've been watching the pymalloc discussions going on recently, and I'm concerned that there may be some fairly major compatibility changes coming.
We are trying to avoid that.
The code has survived essentially unchanged through [Python 1.4] to Python 2.2, and still works fine.
This is a great test case. Can you provide a URL to the code?
It seems to me that there are two options open for me:
1. The Python API from 1.4 is still guaranteed to work unchanged (even given the liberties mentioned above). In that case, do nothing. 2. There are changes required. In that case, make them, protected by python API checks (annoying ifdef fun).
As long as you don't use free() to deallocate objects then you are safe. Python 1.4 has PyMem_DEL. You can use a macro like this: #if PY_VERSION_HEX < 0x01060000 #define PyObject_New PyObject_NEW #define PyObject_Del PyMem_DEL #endif and use PyObject_New and PyObject_Del to manage object memory. Neil
Neil Schemenauer wrote:
You can use a macro like this:
#if PY_VERSION_HEX < 0x01060000 #define PyObject_New PyObject_NEW #define PyObject_Del PyMem_DEL #endif
I just realized that this may be unclear. You don't _need_ to use the above macro to make your code work across versions. As long as you use PyMem_DEL and not free() to deallocate objects the code will work. Neil
On Thu, Apr 04, 2002, Moore, Paul wrote:
OK, so maybe it's time to give up on Python 1.4 compatibility. But ideally, I'd prefer to go for an approach which leaves the code used under Python 2.2 and earlier completely unchanged, and only adds the minimum necessary for newer versions.
Longer term, it might be worth considering upgrading the code to use some of the newer features in the Python API, but in all honesty that's not a great priority for me (and unlikely to be even remotely on the cards until I can realistically desupport everything before 2.0...)
Why can't you realistically desupport pre-2.0? Believe me, I understand all the arguments in favor of keeping support for old versions, but someone who's getting a *new* version of vim has IMO demonstrably indicated that zie wishes to get access to new software. Even if zie needs to support old versions of Python, why shouldn't zie use a current version as the macro language in vim? What I'd suggest doing for your case is targetting Python 2.2 without pyMalloc, and just making sure you follow the 2.3 rules. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are times when effort is important and necessary, but this should not be taken as any kind of moral imperative." --jdecker
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, Apr 05, 2002, Guido van Rossum wrote:
Aahz:
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST.
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default? The default Python for Mandrake 8.1 (*not* the most current version), for example, is Python 2.1.1. I don't think Python 1.5.2 is the default Python for anything other than Red Hat at the moment, and even if I'm wrong about that now, it certainly will be true by the time any new version of vim becomes an installed default. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are times when effort is important and necessary, but this should not be taken as any kind of moral imperative." --jdecker
Aahz wrote:
On Fri, Apr 05, 2002, Guido van Rossum wrote:
Aahz:
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST.
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
The default Python for Mandrake 8.1 (*not* the most current version), for example, is Python 2.1.1. I don't think Python 1.5.2 is the default Python for anything other than Red Hat at the moment, and even if I'm wrong about that now, it certainly will be true by the time any new version of vim becomes an installed default.
Python 1.5.2 is still in active use. Some of the extension writers simply can't drop 1.5.2 because of an active user base still working with it (either because they are using commercial products built on top of 1.5.2 or because they like the speed... Python 2.x is slower than 1.5.2, or because their books still reference the old version). Python's release schedule has changed dramatically over the last few years -- I would expect that the Python 2.0 user base will also last a few more years. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
On Fri, Apr 05, 2002, M.-A. Lemburg wrote:
Aahz wrote:
On Fri, Apr 05, 2002, Guido van Rossum wrote:
Aahz:
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST.
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
The default Python for Mandrake 8.1 (*not* the most current version), for example, is Python 2.1.1. I don't think Python 1.5.2 is the default Python for anything other than Red Hat at the moment, and even if I'm wrong about that now, it certainly will be true by the time any new version of vim becomes an installed default.
Python 1.5.2 is still in active use. Some of the extension writers simply can't drop 1.5.2 because of an active user base still working with it (either because they are using commercial products built on top of 1.5.2 or because they like the speed... Python 2.x is slower than 1.5.2, or because their books still reference the old version).
Python's release schedule has changed dramatically over the last few years -- I would expect that the Python 2.0 user base will also last a few more years.
Yes, yes, yes, yes, yes, yes, yes -- but why is this a problem for vim? I am not talking about some abstract general case; it looks to me that in this specific case backward compatibility isn't an issue, and I still have not seen any explanation for why I'm wrong. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are times when effort is important and necessary, but this should not be taken as any kind of moral imperative." --jdecker
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST.
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
I wasn't talking about vim, sorry. But we should support Python developers who want to write extensions that work with all versions from Python 1.5.2 onwards.
The default Python for Mandrake 8.1 (*not* the most current version), for example, is Python 2.1.1. I don't think Python 1.5.2 is the default Python for anything other than Red Hat at the moment, and even if I'm wrong about that now, it certainly will be true by the time any new version of vim becomes an installed default.
Vendors can release new versions as much as they want, but most users will only upgrade when forced. Python 1.5.2 binaries are still *very* common. I still get mail from people who say "why do you change Python so much? 1.5.2 works just fine for me." --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, Apr 05, 2002, Guido van Rossum wrote:
Why can't you realistically desupport pre-2.0?
Because 1.5.2 is still most commonly found as the default Python, like it or not. I don't care for 1.4, but 1.5.2 support is a MUST.
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
I wasn't talking about vim, sorry. But we should support Python developers who want to write extensions that work with all versions from Python 1.5.2 onwards.
With that I can whole-heartedly agree (after all, I'm writing the BCD module to be backward compatible, and I had to muck with some of Tim's code to do it ;-).
The default Python for Mandrake 8.1 (*not* the most current version), for example, is Python 2.1.1. I don't think Python 1.5.2 is the default Python for anything other than Red Hat at the moment, and even if I'm wrong about that now, it certainly will be true by the time any new version of vim becomes an installed default.
Vendors can release new versions as much as they want, but most users will only upgrade when forced. Python 1.5.2 binaries are still *very* common. I still get mail from people who say "why do you change Python so much? 1.5.2 works just fine for me."
<nod> I just think the criteria should be a bit different for developer tools than end-user products, particularly in the context of an upgraded developer tool being dependent on a new version of Python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are times when effort is important and necessary, but this should not be taken as any kind of moral imperative." --jdecker
[Aahz]
... I am not talking about some abstract general case; it looks to me that in this specific case backward compatibility isn't an issue, and I still have not seen any explanation for why I'm wrong.
Well, give us a reason to believe you're right <wink>. Specific cases can't be decided on "general principles" -- the only way to know what vim's needs actually are is to study its source code. Have you done that? I haven't.
On Fri, Apr 05, 2002, Tim Peters wrote:
[Aahz]
... I am not talking about some abstract general case; it looks to me that in this specific case backward compatibility isn't an issue, and I still have not seen any explanation for why I'm wrong.
Well, give us a reason to believe you're right <wink>. Specific cases can't be decided on "general principles" -- the only way to know what vim's needs actually are is to study its source code. Have you done that? I haven't.
You'll note that I started this exchange by asking why pre-2.0 was a requirement, with none of the responses directly addressing my question. I'm certainly not competent to analyze the code, but I think I can at least ask interesting questions. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "There are times when effort is important and necessary, but this should not be taken as any kind of moral imperative." --jdecker
Aahz <aahz@pythoncraft.com> writes:
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
Because requiring this software might violate company policies. For example, Redhat cannot include more recent versions of Python in their distribution 7.x. More precisely, they can (and do), but they cannot have other packages depend on Python 2.x. They might be able to upgrade vim versions, though (provided those were compatible with the one included in 7.0). If more recent vim versions require more recent Python versions, they cannot upgrade vim. Now, if package xyz required a more recent vim version, they couldn't upgrade that package, either. Regards, Martin
martin wrote:
Aahz <aahz@pythoncraft.com> writes:
<scratch head> I must be missing something here. We're not talking about an end-user product, we're talking about a developer tool. What's wrong with requiring a Python download for someone who at this point wants to *upgrade* vim from the system default?
Because requiring this software might violate company policies. For example, Redhat cannot include more recent versions of Python in their distribution 7.x.
what martin says should be obvious, of course -- people use Python in many different environments, and most of them are quite different from your average python-dev'ers development box. what I don't really understand why this discussion keeps going. isn't it up to the VIM maintainers to decide what requirements their customers might be willing to live with? (if you want me to tell you what I really think, ask me to force PIL users to upgrade to the latest Python version if they want to use the latest PIL version... or post a marketing PEP ;-) </F>
participants (8)
-
Aahz -
Fredrik Lundh -
Guido van Rossum -
M.-A. Lemburg -
martin＠v.loewis.de -
Moore, Paul -
Neil Schemenauer -
Tim Peters