At the LSB meeting, Jeff Licquia asked whether Python could provide binary compatibility with previous versions by use of ELF symbol versioning. In ELF symbol versioning, you can have multiple definitions for the same symbol; clients (e.g. extension modules) would refer to a specific version. During static linking, the most recent (?) version of the symbol is coded into the client object. With symbol versioning, you can change the implementation of a function and even its interface, and it will be compatible as long as you keep the original version around. My first reaction is that this is difficult due to the usage of function-like macros. However, if we replaced those with C functions (perhaps has a compile-time choice), and if we would also hide the layout of structures, I think providing binary compatibility (with a certain baseline version, or multiple of these) would be possible. Of course, several things need to be considered, e.g. - making Py_INCREF/Py_DECREF functions is likely a bad idea for performance reasons. OTOH, it seems safe that Py_INCREF/Py_DECREF can remain as-is for the rest of 2.x. - hiding PyTypeObject is a bad idea for source compatibility. OTOH, we already try to make only compatible changes to it (except when we don't :-), so exposing this as stable ABI might actually work. - certain kinds of modules likely need to be ruled out, e.g. modules that extend the layout of existing types (it would fail to compile because the base struct becomes an incomplete type). All in all, I think providing binary compatibility would be feasible, and should be attempted. What do you think? Regards, Martin
On 12/4/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:
At the LSB meeting, Jeff Licquia asked whether Python could provide binary compatibility with previous versions by use of ELF symbol versioning. In ELF symbol versioning, you can have multiple definitions for the same symbol; clients (e.g. extension modules) would refer to a specific version. During static linking, the most recent (?) version of the symbol is coded into the client object.
With symbol versioning, you can change the implementation of a function and even its interface, and it will be compatible as long as you keep the original version around.
[in a separate message that Python 2.4 is the first targetted version]
All in all, I think providing binary compatibility would be feasible, and should be attempted. What do you think?
Let's assume that 2.4 is the first LSB version. The ABI is different for 2.4 and 2.5. We can't change the ABI for 2.5 since it's already released and our policy is to keep it constant. Where does that leave us for moving forward? Would we have to modify 2.5 to make it entirely compatible with 2.4? Can we skip 2.5 altogether and worry about making 2.6 compatible with 2.4? Even if so, that means no matter what we still need to make 2.4 compatible APIs going forward, right? That will require some archeaology to determine preciesly what we changed from 2.4. Is our current practice of the following change acceptable? Original: PyAPI_FUNC(xxx) Py_Foo(xxx); New version: PyAPI_FUNC(xxx) Py_FooEx(xxx, yyy); #define Py_Foo(x) Py_FooEx(x, NULL) And similarly in the C file, but keeping the original PyFoo with something like: xxx Py_FooEx(...) { ... } #undef Py_Foo xxx Py_Foo(xxx) { return Py_FooEx(xxx, NULL); } I'm not arguing against the LSB at all. I think it's a good thing, but I'm trying to understand what work needs to be done and how we can acheive it in the face of apparent competing requirements. I also realize this is a one time cost and we don't need to address the details right now. n
>> All in all, I think providing binary compatibility would be feasible, >> and should be attempted. What do you think? Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since Neal> it's already released and our policy is to keep it constant. It seems that adhering to LSB's constraints is going to create a new set of problems for Python development. It's unclear to me what LSB brings to Python other than a bunch of new headaches. Skip
Neal Norwitz schrieb:
All in all, I think providing binary compatibility would be feasible, and should be attempted. What do you think?
Let's assume that 2.4 is the first LSB version. The ABI is different for 2.4 and 2.5. We can't change the ABI for 2.5 since it's already released and our policy is to keep it constant.
I'm not absolutely sure it's true, because it may be possible to integrate the 2.4 ABI into future 2.5 releases by means of symbol versioning, but, more likely, that's not possible (as you likely can't represent the difference in symbol versions).
Where does that leave us for moving forward? Would we have to modify 2.5 to make it entirely compatible with 2.4? Can we skip 2.5 altogether and worry about making 2.6 compatible with 2.4?
The most likely outcome is that LSB 3.2 won't provide any Python ABI, but just a Python API (for a scripting language, that's enough to achieve portability of applications). With that, extension modules would not be portable across releases. Then, with LSB 4.0, we could try to specify an ABI as well. Whether that would be the 2.4 ABI, the 2.5 ABI, or the 2.6 ABI must be decided.
Even if so, that means no matter what we still need to make 2.4 compatible APIs going forward, right? That will require some archeaology to determine preciesly what we changed from 2.4.
The LSB people are happy if we declare a subset of the ABI as stable. So if PyFrame_New wasn't in the ABI, that would probably be just fine. I think most ABI breakages are due to changes to structure layout, and we should remove all reliance on structure layout out of the ABI - requiring that ABI-compliant extensions use accessor functions for everything.
Is our current practice of the following change acceptable?
Original: PyAPI_FUNC(xxx) Py_Foo(xxx);
New version: PyAPI_FUNC(xxx) Py_FooEx(xxx, yyy); #define Py_Foo(x) Py_FooEx(x, NULL)
And similarly in the C file, but keeping the original PyFoo with something like:
xxx Py_FooEx(...) { ... } #undef Py_Foo xxx Py_Foo(xxx) { return Py_FooEx(xxx, NULL); }
This would be perfectly fine for ABI stability. They could even accept if we just changed the signature of Py_Foo, and use symbol versioning to provide the old definition (*) - but for API evolution, that wouldn't work, of course. It would mean that extension writers need the old header files to build extensions. The LSB people would be fine with that - they provide a complete set of header files to build LSB-compatible binaries, anyway (so they can tell people: use these headers, and your binaries will be ABI-compliant). Regards, Martin (*) symbol versioning would help in cases where we change a parameter to be of type Py_ssize_t now: we'd provide another definition (with an old version number) that has the original signature, and do the same forwarding.
skip@pobox.com schrieb:
>> All in all, I think providing binary compatibility would be feasible, >> and should be attempted. What do you think?
Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since Neal> it's already released and our policy is to keep it constant.
It seems that adhering to LSB's constraints is going to create a new set of problems for Python development. It's unclear to me what LSB brings to Python other than a bunch of new headaches.
I won't try to defend it, but would suggest that an evaluation is deferred until it is clear what the actual problems are, and then to judge whether they are additional problems (or perhaps just a tightening of procedures which we had been following all along). In any case, having Python in the LSB means that ISVs (software vendors) who target LSB (rather than targetting specific Linux distributions) could develop their applications also in Python (whereas now they have to use C or C++). With not even Java being included, that would put Python into a unique position in that field (perhaps along with Perl, which is also targetted for LSB 3.2 - this wasn't discussed yesterday, basically because nobody from the Perl community was present). Regards, Martin
Martin v. Löwis wrote:
In any case, having Python in the LSB means that ISVs (software vendors) who target LSB (rather than targetting specific Linux distributions) could develop their applications also in Python (whereas now they have to use C or C++).
... without having to include a Python runtime. </F>
"Martin" == Martin v Löwis <martin@v.loewis.de> writes:
Martin> skip@pobox.com schrieb: >> >> All in all, I think providing binary compatibility would be feasible, >> >> and should be attempted. What do you think? >> Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5 since Neal> it's already released and our policy is to keep it constant. >> >> It seems that adhering to LSB's constraints is going to create a new set of >> problems for Python development. It's unclear to me what LSB brings to >> Python other than a bunch of new headaches. Martin> I won't try to defend it, but would suggest that an evaluation Martin> is deferred until it is clear what the actual problems are, and Martin> then to judge whether they are additional problems (or perhaps Martin> just a tightening of procedures which we had been following all Martin> along). Taking one example from this thread, Python's bytecode has always been an internal implementation detail. If I read the thread correctly there is at least a request (if not a requirement) to make it part of an external ABI if Python is to become part of the ABI. That may or may not be a large technical challenge, but I think it would be a significant philosophical change. Martin> In any case, having Python in the LSB means that ISVs (software Martin> vendors) who target LSB (rather than targetting specific Linux Martin> distributions) could develop their applications also in Python Martin> (whereas now they have to use C or C++). Why? Lots of people write portable Python programs today. Skip
On 12/5/06, skip@pobox.com <skip@pobox.com> wrote:
"Martin" == Martin v Löwis <martin@v.loewis.de> writes:
Martin> skip@pobox.com schrieb: >> >> All in all, I think providing binary compatibility would be feasible, >> >> and should be attempted. What do you think? >> Neal> Let's assume that 2.4 is the first LSB version. The ABI is Neal> different for 2.4 and 2.5. We can't change the ABI for 2.5since Neal> it's already released and our policy is to keep it constant. >> >> It seems that adhering to LSB's constraints is going to create a new set of >> problems for Python development. It's unclear to me what LSB brings to >> Python other than a bunch of new headaches.
Martin> I won't try to defend it, but would suggest that an evaluation Martin> is deferred until it is clear what the actual problems are, and Martin> then to judge whether they are additional problems (or perhaps Martin> just a tightening of procedures which we had been following all Martin> along).
Taking one example from this thread, Python's bytecode has always been an internal implementation detail. If I read the thread correctly there is at least a request (if not a requirement) to make it part of an external ABI if Python is to become part of the ABI. That may or may not be a large technical challenge, but I think it would be a significant philosophical change.
I don't think we are being asked to standardize the bytecode, but that we will accept .pyc files as generated by 2.4 in future interpreters as legitimate. That seems to implicitly require us to standardize the bytecode *and* they .pyc file format. -Brett
Could backwards compatibility concerns be addressed by including more than one version of Python in the LSB? Python already allows multiple versions to coexist, so applications targeting the LSB would just need to be explicit about which version of the interpreter to launch. -- Greg
skip@pobox.com schrieb:
Taking one example from this thread, Python's bytecode has always been an internal implementation detail.
I think that has stopped being true at least since I wrote 1997, or perhaps even since dis.py was written (i.e. right from the beginning of the language). With dis.py, the bytecode instructions are documented, and their numeric values accessible through opcode.h. What *hasn't* been documented is the marshal format, as it may change across releases. That hasn't stopped people from using the format as if it were specified, e.g. there is a Perl reader for marshal data: http://search.cpan.org/~simonw/Python-Serialise-Marshal-0.02/lib/Python/Seri...
If I read the thread correctly there is at least a request (if not a requirement) to make it part of an external ABI if Python is to become part of the ABI. That may or may not be a large technical challenge, but I think it would be a significant philosophical change.
The philosophical change would be that a specific version of that would be standardized. The "current" version could still evolve in a less documented way.
Martin> In any case, having Python in the LSB means that ISVs (software Martin> vendors) who target LSB (rather than targetting specific Linux Martin> distributions) could develop their applications also in Python Martin> (whereas now they have to use C or C++).
Why? Lots of people write portable Python programs today.
But if they distribute them, they either have to request that a Python interpreter is installed on the target system, or they have to ship a Python interpreter with their installation DVD (as they can't rely on Python being available on the system, in a certain version). With LSB specifying that Python must be present, and must have a certain (minimal) version, they can include Python scripts in their application, and they will work out of the box without the administrator of the target machine having to do anything. Regards, Martin
Greg Ewing schrieb:
Could backwards compatibility concerns be addressed by including more than one version of Python in the LSB? Python already allows multiple versions to coexist, so applications targeting the LSB would just need to be explicit about which version of the interpreter to launch.
Yes. However, for that to work, the Linux distributors would not want to maintain the old Python releases themselves. Given the LSB release cycle, we would have still to support Python 2.0 today if that would have been included at the time of its release. Of course, for those vendors, *only* security patches matter. They can justify carrying outdated software around for compatibility only, but they cannot justify including it if it has known security bugs. Regards, Martin
participants (6)
-
"Martin v. Löwis" -
Brett Cannon -
Fredrik Lundh -
Greg Ewing -
Neal Norwitz -
skip＠pobox.com