Proposing new file-like object methods
I would like to see few methods added to file() objects: - read(size, *, offset=None) so os.pread() be used going around file description position - write(data, *, offset=None) analog os.pwrite() - flush(sync=False) so os.fsync(f.fileno()) can optionally follow For text files, offset would probably be not supported or count in characters. New methods could be added like readp/writep/fsync instead of modifying current methods, its debatable. Please tell me how to go about pushing this mainstream.
On Aug 1, 2016 7:13 PM, "Arek Bulski" <arek.bulski@gmail.com> wrote:
I would like to see few methods added to file() objects:
- read(size, *, offset=None) so os.pread() be used going around
file description position https://docs.python.org/3/library/os.html#os.pread (Unix) file.pread?
- write(data, *, offset=None) analog os.pwrite()
https://docs.python.org/3/library/os.html#os.pwrite (Unix) file.pwrite?
- flush(sync=False) so os.fsync(f.fileno()) can optionally
follow https://docs.python.org/3/library/os.html#os.fsync (Unix, Windows) file.fsync?
For text files, offset would probably be not supported or count in
characters. New methods could be added like readp/writep/fsync instead of modifying current methods, its debatable. IDK how this works when wrapped with https://docs.python.org/2/library/codecs.html#codecs.open ?
Please tell me how to go about pushing this mainstream.
It's often easier to build a package with the new methods: - e.g. https://pypi.python.org/pypi/backports/1.0 - https://pypi.python.org/pypi/pathlib/1.0.1 https://pypi.python.org/pypi/path.py
On Mon, Aug 1, 2016, at 20:27, Wes Turner wrote:
file.pread?
Wouldn't that be rather like having os.fstatat? For the concept in general, I'm concerned that the equivalent functionality on windows may require that the I/O be done asynchronously, or that the file have been opened in a special way. Any Windows expert care to comment? Also, I don't think windows has pread [etc] functions. Will integrating support for it into high-level files require us to implement our own read/write logic independent of msvcrt? How does this interact with files that may have been opened in text mode via os.open or msvcrt.setmode; or do we not care about compatibility for this case? Should O_TEXT and msvcrt.setmode be deprecated?
On Tue, Aug 2, 2016 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
Updating the file position is all or nothing. In synchronous mode, which is what the CRT's low I/O implementation uses, the file position is always updated. However, one can atomically set the file position before a read by passing an overlapped with an offset to ReadFile, which in turn sets the ByteOffset argument of the underlying NtReadFile [1] system call. [1]: https://msdn.microsoft.com/en-us/library/ff567072 For example: import _winapi import ctypes from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) ULONG_PTR = wintypes.WPARAM class OVERLAPPED(ctypes.Structure): class _OFF_PTR(ctypes.Union): class _OFFSET(ctypes.Structure): _fields_ = (('Offset', wintypes.DWORD), ('OffsetHigh', wintypes.DWORD)) _fields_ = (('_offset', _OFFSET), ('Pointer', wintypes.LPVOID)) _anonymous_ = ('_offset',) _fields_ = (('Internal', ULONG_PTR), ('InternalHigh', ULONG_PTR), ('_off_ptr', _OFF_PTR), ('hEvent', wintypes.HANDLE)) _anonymous_ = ('_off_ptr',) with open('test.txt', 'wb') as f: f.write(b'0123456789') h = _winapi.CreateFile('test.txt', 0x80000000, 3, 0, 3, 0, 0) buf = (ctypes.c_char * 5)() n = (wintypes.DWORD * 1)() fp = (wintypes.LARGE_INTEGER * 1)() ov = (OVERLAPPED * 1)() ov[0].Offset = 5 >>> kernel32.ReadFile(h, buf, 5, n, ov) 1 >>> buf[:] b'56789' >>> kernel32.SetFilePointerEx(h, 0, fp, 1) 1 >>> fp[0] 10 Note that after reading 5 bytes at an offset of 5, the current file position is 10. With pread() it would still be at 0. The file position can be tracked and set atomically before each read or write, but I don't think this is practical unless it's part of a larger project to use the Windows API directly.
On Aug 1, 2016 7:13 PM, "Arek Bulski" <arek.bulski@gmail.com> wrote:
I would like to see few methods added to file() objects:
- read(size, *, offset=None) so os.pread() be used going around
file description position https://docs.python.org/3/library/os.html#os.pread (Unix) file.pread?
- write(data, *, offset=None) analog os.pwrite()
https://docs.python.org/3/library/os.html#os.pwrite (Unix) file.pwrite?
- flush(sync=False) so os.fsync(f.fileno()) can optionally
follow https://docs.python.org/3/library/os.html#os.fsync (Unix, Windows) file.fsync?
For text files, offset would probably be not supported or count in
characters. New methods could be added like readp/writep/fsync instead of modifying current methods, its debatable. IDK how this works when wrapped with https://docs.python.org/2/library/codecs.html#codecs.open ?
Please tell me how to go about pushing this mainstream.
It's often easier to build a package with the new methods: - e.g. https://pypi.python.org/pypi/backports/1.0 - https://pypi.python.org/pypi/pathlib/1.0.1 https://pypi.python.org/pypi/path.py
On Mon, Aug 1, 2016, at 20:27, Wes Turner wrote:
file.pread?
Wouldn't that be rather like having os.fstatat? For the concept in general, I'm concerned that the equivalent functionality on windows may require that the I/O be done asynchronously, or that the file have been opened in a special way. Any Windows expert care to comment? Also, I don't think windows has pread [etc] functions. Will integrating support for it into high-level files require us to implement our own read/write logic independent of msvcrt? How does this interact with files that may have been opened in text mode via os.open or msvcrt.setmode; or do we not care about compatibility for this case? Should O_TEXT and msvcrt.setmode be deprecated?
On Tue, Aug 2, 2016 at 2:00 AM, Random832 <random832@fastmail.com> wrote:
Updating the file position is all or nothing. In synchronous mode, which is what the CRT's low I/O implementation uses, the file position is always updated. However, one can atomically set the file position before a read by passing an overlapped with an offset to ReadFile, which in turn sets the ByteOffset argument of the underlying NtReadFile [1] system call. [1]: https://msdn.microsoft.com/en-us/library/ff567072 For example: import _winapi import ctypes from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) ULONG_PTR = wintypes.WPARAM class OVERLAPPED(ctypes.Structure): class _OFF_PTR(ctypes.Union): class _OFFSET(ctypes.Structure): _fields_ = (('Offset', wintypes.DWORD), ('OffsetHigh', wintypes.DWORD)) _fields_ = (('_offset', _OFFSET), ('Pointer', wintypes.LPVOID)) _anonymous_ = ('_offset',) _fields_ = (('Internal', ULONG_PTR), ('InternalHigh', ULONG_PTR), ('_off_ptr', _OFF_PTR), ('hEvent', wintypes.HANDLE)) _anonymous_ = ('_off_ptr',) with open('test.txt', 'wb') as f: f.write(b'0123456789') h = _winapi.CreateFile('test.txt', 0x80000000, 3, 0, 3, 0, 0) buf = (ctypes.c_char * 5)() n = (wintypes.DWORD * 1)() fp = (wintypes.LARGE_INTEGER * 1)() ov = (OVERLAPPED * 1)() ov[0].Offset = 5 >>> kernel32.ReadFile(h, buf, 5, n, ov) 1 >>> buf[:] b'56789' >>> kernel32.SetFilePointerEx(h, 0, fp, 1) 1 >>> fp[0] 10 Note that after reading 5 bytes at an offset of 5, the current file position is 10. With pread() it would still be at 0. The file position can be tracked and set atomically before each read or write, but I don't think this is practical unless it's part of a larger project to use the Windows API directly.
participants (4)
-
Arek Bulski -
eryk sun -
Random832 -
Wes Turner