OpenVMS file system and UNIVERSAL_NEWLINES support
Hi all, I am one of the maintainer of Python on OpenVMS. Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed. This is a major problem on OpenVMS. VMS has a complex file system which is mostly record oriented and not stream oriented (even if it support stream oriented files, this is not the default). For example it support the following record formats: fixed-length, variable-length, variable with fixed-length control, 3 stream format, etc... The default is variable-length. So reading (opening) text file, for example during a import, in binary mode lead to incorrect results because there are no '\n' or any character (or combinaison of characters) at the end of the line. If the file is not open in binary mode the VMS CRTL automatically append a '\n' at the end of the record, so all work correctly. This not the case in binary mode, in this case the record is read as is, in fact you even get the control part of the record. So, is it possible to maintain this test, even as undocumented or is it acceptable to transform it into a specific VMS test. But I suspect that other mainframe file system will have the same problem. Thanks for any help. Regards, Jean-François
On Mon, Feb 23, 2004, Jean-François Piéronne wrote:
Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed.
This is a major problem on OpenVMS.
Please file a bug report on SF so that this doesn't get lost, then post the bug number here. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Do not taunt happy fun for loops. Do not change lists you are looping over." --Remco Gerlich, comp.lang.python
Aahz wrote:
On Mon, Feb 23, 2004, Jean-François Piéronne wrote:
Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed.
This is a major problem on OpenVMS.
Please file a bug report on SF so that this doesn't get lost, then post the bug number here.
Done, request ID: 903339 Thanks, Jean-François
On Mon, Feb 23, 2004, Jean-François Piéronne wrote:
Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed.
This is a major problem on OpenVMS.
Please file a bug report on SF so that this doesn't get lost, then post the bug number here.
I have submit a fix for this problem, bug 903339 Cheers, Jean-François
>>> Building from time to time Python 2.4 from CVS snapshot, I have just >>> noticed that all the conditional compilation against >>> WITH_UNIVERSAL_NEWLINES has been removed. >> Please file a bug report on SF so that this doesn't get lost, then >> post the bug number here. Jean-François> I have submit a fix for this problem, bug 903339 I'll try to get to this in the next couple days. I believe I saved the discussion of possible solutions. Skip
On 23-feb-04, at 18:15, Jean-François Piéronne wrote:
Hi all,
I am one of the maintainer of Python on OpenVMS.
Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed.
This is a major problem on OpenVMS.
Ah, I was afraid this was going to happen when I saw the checkin message about getting rid of the universal newlines conditional. The easiest solution (apart from putting WITH_UNIVERSAL_NEWLINES back in) is to just leave all the universal newlines machinery in place, but disable it. So, where the old situation was that there were ifdefs all over the place the new situation would be that all the code always goes through Py_UniversalNewlineFgets() and Py_UniversalNewlineFread(), but on systems with record-based I/O these just call fgets() and fread(). Then there's only one more ifdef to go: in open_the_file() don't set the mode to "rb" but just plain "r". I assume that sockets and other places where WITH_UNIVERSAL_NEWLINES may have been referred to aren't going to be a problem for you because those aren't applicable to VMS anyway, right? -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
Jack Jansen wrote:
On 23-feb-04, at 18:15, Jean-François Piéronne wrote:
Hi all,
I am one of the maintainer of Python on OpenVMS.
Building from time to time Python 2.4 from CVS snapshot, I have just noticed that all the conditional compilation against WITH_UNIVERSAL_NEWLINES has been removed.
This is a major problem on OpenVMS.
Ah, I was afraid this was going to happen when I saw the checkin message about getting rid of the universal newlines conditional.
The easiest solution (apart from putting WITH_UNIVERSAL_NEWLINES back in) is to just leave all the universal newlines machinery in place, but disable it. So, where the old situation was that there were ifdefs all over the place the new situation would be that all the code always goes through Py_UniversalNewlineFgets() and Py_UniversalNewlineFread(), but on systems with record-based I/O these just call fgets() and fread(). Then there's only one more ifdef to go: in open_the_file() don't set the mode to "rb" but just plain "r".
I assume that sockets and other places where WITH_UNIVERSAL_NEWLINES may have been referred to aren't going to be a problem for you because those aren't applicable to VMS anyway, right? -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
thanks for your help, Before I submit a patch, I have done the following update which seem to fix the problem, any comment are welcome. patch for include/fileobject.h @@ -54,3 +54,7 @@ */ +#ifdef __VMS +#define PY_STDIOTEXTMODE "" +#else #define PY_STDIOTEXTMODE "b" +#endif char *Py_UniversalNewlineFgets(char *, int, FILE*, PyObject *); patch for objects/fileobject.c @@ -119,3 +124,7 @@ f->f_buf = NULL; +#ifdef __VMS + f->f_univ_newline = 0; +#else f->f_univ_newline = (strchr(mode, 'U') != NULL); +#endif f->f_newlinetypes = NEWLINE_UNKNOWN; @@ -164,3 +173,11 @@ if (strcmp(mode, "U") == 0 || strcmp(mode, "rU") == 0) +#ifdef __VMS + /* Compatibility: specifying U in a Python without universal + ** newlines is allowed, and the file is opened as a normal text + ** file. + */ + mode = "r"; +#else mode = "rb"; +#endif #ifdef MS_WINDOWS patch for modules/bz2modules.c @@ -1293,3 +1293,7 @@ case 'U': +#ifdef __VMS + self->f_univ_newline = 0; +#else self->f_univ_newline = 1; +#endif break; Cheers, Jean-François
On 25 Feb 2004, at 17:39, Jean-François Piéronne wrote:
Before I submit a patch, I have done the following update which seem to fix the problem, any comment are welcome.
patch for include/fileobject.h @@ -54,3 +54,7 @@ */ +#ifdef __VMS +#define PY_STDIOTEXTMODE "" +#else #define PY_STDIOTEXTMODE "b" +#endif
[...] I wouldn't call it __VMS: if Python still runs on OS/390 or MVS or whatever they'll have the same problem. And I/O system where the underlying filesystem isn't bytestream-based is really what we're talking about, I think. -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
Jack Jansen wrote:
On 25 Feb 2004, at 17:39, Jean-François Piéronne wrote:
Before I submit a patch, I have done the following update which seem to fix the problem, any comment are welcome.
patch for include/fileobject.h @@ -54,3 +54,7 @@ */ +#ifdef __VMS +#define PY_STDIOTEXTMODE "" +#else #define PY_STDIOTEXTMODE "b" +#endif
[...]
I wouldn't call it __VMS: if Python still runs on OS/390 or MVS or whatever they'll have the same problem. And I/O system where the underlying filesystem isn't bytestream-based is really what we're talking about, I think.
Correct, none of these patches are OpenVMS specific. Any conditional name suggestion is welcome... Someone has also suggest to simplified #ifdef __VMS /* Compatibility: specifying U in a Python without universal ** newlines is allowed, and the file is opened as a normal text ** file. */ mode = "r"; #else mode = "rb"; #endif to mode = "r" PY_STDIOTEXTMODE; Thanks for your help. Jean-François
>> I wouldn't call it __VMS: if Python still runs on OS/390 or MVS or >> whatever they'll have the same problem. And I/O system where the >> underlying filesystem isn't bytestream-based is really what we're >> talking about, I think. Jean-François> Correct, none of these patches are OpenVMS specific. Jean-François> Any conditional name suggestion is welcome... What about Py_HAS_STREAM_IO? It would be defined by default. On platforms like OpenVMS, you could #undef it in pyport.h or through some other mechanism. Or the contr, something like Py_HAS_NO_STREAM_IO. That doesn't read very well though. Skip
On Fri, Feb 27, 2004, Skip Montanaro wrote:
What about Py_HAS_STREAM_IO? It would be defined by default. On platforms like OpenVMS, you could #undef it in pyport.h or through some other mechanism.
Or the contr, something like Py_HAS_NO_STREAM_IO. That doesn't read very well though.
Py_HAS_RECORD_IO -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Do not taunt happy fun for loops. Do not change lists you are looping over." --Remco Gerlich, comp.lang.python
Aahz wrote:
On Fri, Feb 27, 2004, Skip Montanaro wrote:
What about Py_HAS_STREAM_IO? It would be defined by default. On platforms like OpenVMS, you could #undef it in pyport.h or through some other mechanism.
Or the contr, something like Py_HAS_NO_STREAM_IO. That doesn't read very well though.
Py_HAS_RECORD_IO
What about Py_USE_RECORD_IO which would need to be defined only on VMS and probably a few others platforms and is undefined by default. What is the simplest method for you? Cheers, Jean-François
participants (4)
-
Aahz -
Jack Jansen -
Jean-François Piéronne -
Skip Montanaro