[Python-checkins] r70152 - in python/branches/py3k: Doc/library/io.rst Lib/_pyio.py Lib/importlib/__init__.py Lib/importlib/_bootstrap.py Lib/io.py Lib/test/test_bufio.py Lib/test/test_descr.py Lib/test/test_file.py Lib/test/test_fileio.py Lib/test/test_io.py Lib/test/test_largefile.py Lib/test/test_memoryio.py Lib/test/test_univnewlines.py Lib/test/test_uu.py Makefile.pre.in Misc/NEWS Modules/Setup.dist Modules/_bufferedio.c Modules/_bytesio.c Modules/_fileio.c Modules/_iobase.c Modules/_iomodule.h Modules/_stringio.c Modules/_textio.c Modules/io.c PC/VC6/pythoncore.dsp PC/config.c PCbuild/pythoncore.vcproj Python/pythonrun.c setup.py
benjamin.peterson
python-checkins at python.org
Wed Mar 4 01:14:52 CET 2009
Author: benjamin.peterson
Date: Wed Mar 4 01:14:51 2009
New Revision: 70152
Log:
merge the io-c branch: C implementation of the io module
The main io module now uses the C implementation. The Python one still exists
in Lib/_pyio.py for ease of testing new features and usefulness to other
implementers.
The rewrite was done by Antoine Pitrou and Amaury Forgeot d'Arc. I was slightly
helpful at the end. :)
Following are the log messages from the io-c branch:
Merged revisions 68683-68685,68687-68689,68693,68704,68741-68743,68745,68747,68752-68754,68756,68758,68812,68816-68817,68820-68822,68824-68825,68828,68876-68877,69037,69044,69104,69115,69194,69626-69629,69636,69638,69641-69642,69644-69654,69656-69661,69671,69677,69812-69815,69817,69827-69830,69839,69841-69845,69848,69850,69852,69854,69860,69865-69866,69868,69872-69873,69885,69888,69891-69893,69911,69913-69916,69963,70033,70035,70038,70041-70048,70067-70070,70075,70112,70133,70135,70140 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/io-c
........
r68683 | antoine.pitrou | 2009-01-17 17:13:48 -0600 (Sat, 17 Jan 2009) | 3 lines
Merge in changes from the io-c sandbox. Tests will follow in separate commits.
........
r68684 | antoine.pitrou | 2009-01-17 17:17:26 -0600 (Sat, 17 Jan 2009) | 3 lines
Fixes and additions to test_io.py
........
r68685 | antoine.pitrou | 2009-01-17 17:22:04 -0600 (Sat, 17 Jan 2009) | 1 line
Fix test_fileio
........
r68687 | antoine.pitrou | 2009-01-17 17:35:11 -0600 (Sat, 17 Jan 2009) | 3 lines
Add dependency to _iomodule.h for the various C sources
........
r68688 | antoine.pitrou | 2009-01-17 17:38:18 -0600 (Sat, 17 Jan 2009) | 3 lines
These precautions are not needed anymore!
........
r68689 | antoine.pitrou | 2009-01-17 17:41:48 -0600 (Sat, 17 Jan 2009) | 3 lines
Fix another test
........
r68693 | antoine.pitrou | 2009-01-17 17:49:58 -0600 (Sat, 17 Jan 2009) | 3 lines
Fix test_uu (which was using private attributes of TextIOWrapper)
........
r68704 | antoine.pitrou | 2009-01-17 18:45:29 -0600 (Sat, 17 Jan 2009) | 3 lines
Most io sources are Py_ssize_t-clean (I don't know about bytesio and stringio)
........
r68741 | antoine.pitrou | 2009-01-18 15:20:30 -0600 (Sun, 18 Jan 2009) | 3 lines
Check return type in TextIOWrapper.__next__
........
r68742 | antoine.pitrou | 2009-01-18 15:28:48 -0600 (Sun, 18 Jan 2009) | 4 lines
Make binary buffered readline and iteration much faster
(8x as fast as the IOBase generic implementation)
........
r68743 | antoine.pitrou | 2009-01-18 15:47:47 -0600 (Sun, 18 Jan 2009) | 3 lines
Reinsert test_io_after_close (was removed by mistake)
........
r68745 | antoine.pitrou | 2009-01-18 16:16:06 -0600 (Sun, 18 Jan 2009) | 3 lines
Add read, read1 and write methods to BufferedIOBase
........
r68747 | antoine.pitrou | 2009-01-18 16:35:58 -0600 (Sun, 18 Jan 2009) | 3 lines
Kill test failure
........
r68752 | amaury.forgeotdarc | 2009-01-18 17:05:43 -0600 (Sun, 18 Jan 2009) | 3 lines
Fix a segfault when e.g a BufferedReader is created with a FileIO in
read mode.
........
r68753 | antoine.pitrou | 2009-01-18 17:13:09 -0600 (Sun, 18 Jan 2009) | 3 lines
Add truncate() to text IO objects
........
r68754 | antoine.pitrou | 2009-01-18 17:51:08 -0600 (Sun, 18 Jan 2009) | 3 lines
Remove IOBase.__del__ and replace it with custom code with tp_dealloc
........
r68756 | antoine.pitrou | 2009-01-18 18:10:16 -0600 (Sun, 18 Jan 2009) | 3 lines
Remove irrelevant comment.
........
r68758 | antoine.pitrou | 2009-01-18 18:36:16 -0600 (Sun, 18 Jan 2009) | 3 lines
in importlib:_fileio._FileIO -> _io.FileIO
........
r68812 | antoine.pitrou | 2009-01-20 14:15:51 -0600 (Tue, 20 Jan 2009) | 3 lines
Add garbage collection support to FileIO objects
........
r68816 | antoine.pitrou | 2009-01-20 14:56:28 -0600 (Tue, 20 Jan 2009) | 3 lines
Add GC support to Buffered and Text IO objects
........
r68817 | antoine.pitrou | 2009-01-20 15:19:45 -0600 (Tue, 20 Jan 2009) | 3 lines
Add some file headers
........
r68820 | antoine.pitrou | 2009-01-20 15:29:59 -0600 (Tue, 20 Jan 2009) | 3 lines
Add class TextIOBase
........
r68821 | antoine.pitrou | 2009-01-20 15:36:16 -0600 (Tue, 20 Jan 2009) | 3 lines
Add properties to TextIOBase
........
r68822 | antoine.pitrou | 2009-01-20 15:41:19 -0600 (Tue, 20 Jan 2009) | 3 lines
Disable the pure Python TextIOBase class, and inject C the implementation instead
........
r68824 | antoine.pitrou | 2009-01-20 16:36:28 -0600 (Tue, 20 Jan 2009) | 3 lines
Fix two leaks
........
r68825 | antoine.pitrou | 2009-01-20 16:38:29 -0600 (Tue, 20 Jan 2009) | 3 lines
FileIO.name is just a plain attribute, we can set it directly
........
r68828 | antoine.pitrou | 2009-01-20 17:06:33 -0600 (Tue, 20 Jan 2009) | 3 lines
Speed up closed checks on text IO objects. Good for a 25% speedup on small ops.
........
r68876 | antoine.pitrou | 2009-01-23 17:01:25 -0600 (Fri, 23 Jan 2009) | 3 lines
Two typos
........
r68877 | antoine.pitrou | 2009-01-23 18:13:20 -0600 (Fri, 23 Jan 2009) | 3 lines
Remove two unused functions
........
r69037 | amaury.forgeotdarc | 2009-01-27 17:10:25 -0600 (Tue, 27 Jan 2009) | 2 lines
Update the win32 project files
........
r69044 | antoine.pitrou | 2009-01-27 18:51:07 -0600 (Tue, 27 Jan 2009) | 3 lines
Improve heuristic in IncrementalNewlineDecoder + some micro-optimizations
........
r69104 | antoine.pitrou | 2009-01-29 15:23:42 -0600 (Thu, 29 Jan 2009) | 3 lines
Fix some crashers found by Victor
........
r69115 | hirokazu.yamamoto | 2009-01-29 20:36:28 -0600 (Thu, 29 Jan 2009) | 1 line
Updated VC6 project file.
........
r69194 | antoine.pitrou | 2009-02-01 16:57:18 -0600 (Sun, 01 Feb 2009) | 3 lines
Fix downcasting warnings in 32-bit mode with 64-bit offsets (Windows)
........
r69626 | benjamin.peterson | 2009-02-14 17:33:34 -0600 (Sat, 14 Feb 2009) | 1 line
only catch AttributeError and UnsupportedOperation
........
r69627 | benjamin.peterson | 2009-02-14 21:35:28 -0600 (Sat, 14 Feb 2009) | 1 line
give the IO module its own state and store the os and locale modules in it
........
r69628 | benjamin.peterson | 2009-02-14 22:08:32 -0600 (Sat, 14 Feb 2009) | 1 line
put interned strings in the module state structure
........
r69629 | benjamin.peterson | 2009-02-14 22:15:29 -0600 (Sat, 14 Feb 2009) | 1 line
put UnsupportedOperation in the module state
........
r69636 | benjamin.peterson | 2009-02-15 08:31:42 -0600 (Sun, 15 Feb 2009) | 1 line
dealloc unsupported_operation
........
r69638 | benjamin.peterson | 2009-02-15 09:24:45 -0600 (Sun, 15 Feb 2009) | 1 line
actually test the C implementation
........
r69641 | benjamin.peterson | 2009-02-15 10:12:37 -0600 (Sun, 15 Feb 2009) | 5 lines
make interned strings globals again ;(
putting them in the module state was asking for trouble when the module
was dealloced before the classes in it were
........
r69642 | benjamin.peterson | 2009-02-15 10:19:45 -0600 (Sun, 15 Feb 2009) | 1 line
actually test the python implementations
........
r69644 | antoine.pitrou | 2009-02-15 11:59:30 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix memory leak in destructor when a Python class inherits from IOBase (or an IOBase-derived type)
........
r69645 | antoine.pitrou | 2009-02-15 12:23:26 -0600 (Sun, 15 Feb 2009) | 3 lines
Add a warning about the embarassing state of IOBase finalization
........
r69646 | antoine.pitrou | 2009-02-15 13:14:42 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix opening of 8-bit filenames with FileIO
........
r69647 | antoine.pitrou | 2009-02-15 13:20:22 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix leak in FileIO constructor
........
r69648 | antoine.pitrou | 2009-02-15 13:58:16 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix some refleaks
........
r69649 | antoine.pitrou | 2009-02-15 14:05:13 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix a leak in IOBase.writelines
........
r69650 | antoine.pitrou | 2009-02-15 14:11:56 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix memory leak in BufferedWriter.truncate
........
r69651 | antoine.pitrou | 2009-02-15 14:25:34 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix a leak in TextIOWrapper.seek
........
r69652 | antoine.pitrou | 2009-02-15 14:26:28 -0600 (Sun, 15 Feb 2009) | 3 lines
Unify implementations of truncate for buffered objects
........
r69653 | antoine.pitrou | 2009-02-15 15:15:15 -0600 (Sun, 15 Feb 2009) | 3 lines
Fix more leaks in TextIOWrapper
........
r69654 | antoine.pitrou | 2009-02-15 15:21:57 -0600 (Sun, 15 Feb 2009) | 3 lines
Smaller chunk size for a faster test
........
r69656 | benjamin.peterson | 2009-02-15 17:29:48 -0600 (Sun, 15 Feb 2009) | 1 line
braces make this much clearer
........
r69657 | benjamin.peterson | 2009-02-15 17:46:07 -0600 (Sun, 15 Feb 2009) | 1 line
use the correct macro
........
r69658 | antoine.pitrou | 2009-02-15 19:38:59 -0600 (Sun, 15 Feb 2009) | 5 lines
Fix crash in test_urllib2_localnet in debug mode. It was due to an HTTPResponse
object being revived when calling its close() method in IOBase's tp_dealloc.
_PyIOBase_finalize() starts looking scary...
........
r69659 | benjamin.peterson | 2009-02-15 20:55:48 -0600 (Sun, 15 Feb 2009) | 1 line
fix segfault on initialization failing
........
r69660 | benjamin.peterson | 2009-02-15 21:09:31 -0600 (Sun, 15 Feb 2009) | 1 line
apparently locale.getprefferedencoding() can raise a ImportError, too
........
r69661 | benjamin.peterson | 2009-02-15 21:54:15 -0600 (Sun, 15 Feb 2009) | 1 line
it's amazing this worked at all; I was using the wrong structs!
........
r69671 | benjamin.peterson | 2009-02-16 08:38:27 -0600 (Mon, 16 Feb 2009) | 1 line
add garbage collection support to bytesio
........
r69677 | benjamin.peterson | 2009-02-16 10:31:03 -0600 (Mon, 16 Feb 2009) | 5 lines
reduce ImportError catching code duplication
I'm not sure this makes the code clearer with its new gotos, but
at least I added a big fat comment
........
r69812 | antoine.pitrou | 2009-02-20 13:50:16 -0600 (Fri, 20 Feb 2009) | 3 lines
_StringIO now belongs to the _io modules, rather to its own _stringio module
........
r69813 | antoine.pitrou | 2009-02-20 13:58:22 -0600 (Fri, 20 Feb 2009) | 3 lines
Add a test for StringIO properties
........
r69814 | antoine.pitrou | 2009-02-20 14:06:03 -0600 (Fri, 20 Feb 2009) | 3 lines
Reimplement a few trivial StringIO functions and properties in C
........
r69815 | antoine.pitrou | 2009-02-20 14:13:11 -0600 (Fri, 20 Feb 2009) | 3 lines
Add the line_buffering property to TextIOWrapper, and test for it
........
r69817 | antoine.pitrou | 2009-02-20 14:45:50 -0600 (Fri, 20 Feb 2009) | 4 lines
Allow IncrementalNewlineDecoder to take unicode objects as decoding input if the decoder parameter is None
This will help rewriting StringIO to C
........
r69827 | antoine.pitrou | 2009-02-20 19:00:30 -0600 (Fri, 20 Feb 2009) | 3 lines
Rewrite most of StringIO in C. Some almost empty stubs remain to be converted.
........
r69828 | antoine.pitrou | 2009-02-20 19:09:25 -0600 (Fri, 20 Feb 2009) | 3 lines
Plug a leak, and remove an unused string
........
r69829 | benjamin.peterson | 2009-02-20 20:02:28 -0600 (Fri, 20 Feb 2009) | 1 line
this assertions makes more sense here
........
r69830 | benjamin.peterson | 2009-02-20 20:03:04 -0600 (Fri, 20 Feb 2009) | 1 line
PyModule_AddObject can fail; simplify this code with a macro
........
r69839 | antoine.pitrou | 2009-02-21 12:54:01 -0600 (Sat, 21 Feb 2009) | 3 lines
StringIO is now written entirely in C (and blazingly fast)
........
r69841 | benjamin.peterson | 2009-02-21 14:05:40 -0600 (Sat, 21 Feb 2009) | 1 line
split the Python implementation of io into another module and rewrite the tests to test both implementations
........
r69842 | benjamin.peterson | 2009-02-21 14:10:00 -0600 (Sat, 21 Feb 2009) | 1 line
closed is not a function
........
r69843 | benjamin.peterson | 2009-02-21 14:13:04 -0600 (Sat, 21 Feb 2009) | 1 line
fix __all__ test
........
r69844 | benjamin.peterson | 2009-02-21 14:21:24 -0600 (Sat, 21 Feb 2009) | 1 line
fix the rest of the Misc tests
........
r69845 | benjamin.peterson | 2009-02-21 14:26:59 -0600 (Sat, 21 Feb 2009) | 1 line
RawIOBase is better for FileIO
........
r69848 | benjamin.peterson | 2009-02-21 15:33:53 -0600 (Sat, 21 Feb 2009) | 1 line
fix some more tests broken by bag argument validation
........
r69850 | benjamin.peterson | 2009-02-21 16:16:42 -0600 (Sat, 21 Feb 2009) | 1 line
make the python IncrementalNewineDecoder support a None decoder
........
r69852 | benjamin.peterson | 2009-02-21 16:36:09 -0600 (Sat, 21 Feb 2009) | 1 line
fix a BlockingIOError.characters_written bug
........
r69854 | benjamin.peterson | 2009-02-21 16:49:02 -0600 (Sat, 21 Feb 2009) | 1 line
check whence
........
r69860 | benjamin.peterson | 2009-02-21 17:42:50 -0600 (Sat, 21 Feb 2009) | 1 line
fix some of these Misbehaving io tests
........
r69865 | benjamin.peterson | 2009-02-21 18:59:52 -0600 (Sat, 21 Feb 2009) | 1 line
don't use super here()
........
r69866 | benjamin.peterson | 2009-02-21 19:05:28 -0600 (Sat, 21 Feb 2009) | 1 line
use implementation specific classes
........
r69868 | benjamin.peterson | 2009-02-21 22:12:05 -0600 (Sat, 21 Feb 2009) | 1 line
use a more DRY friendly approach to injecting module contents into test classes
........
r69872 | antoine.pitrou | 2009-02-22 13:39:45 -0600 (Sun, 22 Feb 2009) | 3 lines
Sanitize destructor behaviour of IOBase. Now Python-defined attributes can be accessed from close().
........
r69873 | antoine.pitrou | 2009-02-22 13:50:14 -0600 (Sun, 22 Feb 2009) | 4 lines
Only set the internal fd after it has been checked to be valid
(otherwise, the destructor will attempt to close it)
........
r69885 | benjamin.peterson | 2009-02-22 15:30:14 -0600 (Sun, 22 Feb 2009) | 1 line
convert some other tests to use both io implementations
........
r69888 | antoine.pitrou | 2009-02-22 17:03:16 -0600 (Sun, 22 Feb 2009) | 3 lines
Silence all exceptions when finalizing
........
r69891 | benjamin.peterson | 2009-02-22 17:27:24 -0600 (Sun, 22 Feb 2009) | 1 line
convert another test to test both io implementations
........
r69892 | benjamin.peterson | 2009-02-22 17:32:15 -0600 (Sun, 22 Feb 2009) | 1 line
help poor people like me to find their io tests (did I miss any?)
........
r69893 | benjamin.peterson | 2009-02-22 17:37:56 -0600 (Sun, 22 Feb 2009) | 1 line
put a big note in the test telling people to write tests for both implementations now
........
r69911 | antoine.pitrou | 2009-02-23 13:57:18 -0600 (Mon, 23 Feb 2009) | 3 lines
expose DEFAULT_BUFFER_SIZE again (fixes a bunch of test failures)
........
r69913 | antoine.pitrou | 2009-02-23 14:10:30 -0600 (Mon, 23 Feb 2009) | 4 lines
Do the cyclic garbage collection tests only on the C version.
The Python version is helpless as it uses __del__.
........
r69914 | antoine.pitrou | 2009-02-23 14:21:41 -0600 (Mon, 23 Feb 2009) | 3 lines
Adapt test_largefile to test both implementations
........
r69915 | antoine.pitrou | 2009-02-23 14:25:14 -0600 (Mon, 23 Feb 2009) | 3 lines
One small failure
........
r69916 | antoine.pitrou | 2009-02-23 14:28:33 -0600 (Mon, 23 Feb 2009) | 3 lines
Add a comment, at BP's request
........
r69963 | antoine.pitrou | 2009-02-25 09:42:59 -0600 (Wed, 25 Feb 2009) | 3 lines
Add a test of ABC inheritance
........
r70033 | antoine.pitrou | 2009-02-27 15:49:50 -0600 (Fri, 27 Feb 2009) | 3 lines
The base classes now are ABCs.
........
r70035 | benjamin.peterson | 2009-02-27 15:57:41 -0600 (Fri, 27 Feb 2009) | 1 line
good house keeping
........
r70038 | antoine.pitrou | 2009-02-27 17:05:23 -0600 (Fri, 27 Feb 2009) | 4 lines
Make the buffer allocation overflow tests specific to the C implementation, since the Python implementation resizes its buffers when needed rather than allocating them up front.
........
r70041 | benjamin.peterson | 2009-02-27 18:26:12 -0600 (Fri, 27 Feb 2009) | 1 line
kill java naming for sanity
........
r70042 | benjamin.peterson | 2009-02-27 18:28:53 -0600 (Fri, 27 Feb 2009) | 2 lines
timingTest is superseded by iobench
........
r70043 | antoine.pitrou | 2009-02-27 19:13:50 -0600 (Fri, 27 Feb 2009) | 3 lines
Remove the last traces of java naming in test_io
........
r70044 | antoine.pitrou | 2009-02-27 19:18:34 -0600 (Fri, 27 Feb 2009) | 3 lines
Better resource cleanup
........
r70045 | antoine.pitrou | 2009-02-27 19:29:00 -0600 (Fri, 27 Feb 2009) | 3 lines
Remove dubious uses of super(), and fix one test
........
r70046 | antoine.pitrou | 2009-02-27 19:31:00 -0600 (Fri, 27 Feb 2009) | 3 lines
Bump up CHUNK_SIZE (no need to make the Python version look slower than it is)
........
r70047 | benjamin.peterson | 2009-02-27 20:03:26 -0600 (Fri, 27 Feb 2009) | 1 line
fix typo
........
r70048 | benjamin.peterson | 2009-02-27 21:35:11 -0600 (Fri, 27 Feb 2009) | 1 line
move code to a better place
........
r70067 | benjamin.peterson | 2009-02-28 10:43:20 -0600 (Sat, 28 Feb 2009) | 4 lines
1. make sure to undo buffered read aheads in BufferedRandom.seek()
2. refill the buffer if have <= 0
3. fix the last failing test_io test!
........
r70068 | benjamin.peterson | 2009-02-28 10:57:50 -0600 (Sat, 28 Feb 2009) | 1 line
define read1() on the python implementation's BufferedIOBase
........
r70069 | benjamin.peterson | 2009-02-28 11:01:17 -0600 (Sat, 28 Feb 2009) | 1 line
document read1() in BufferedIOBase
........
r70070 | benjamin.peterson | 2009-02-28 11:06:42 -0600 (Sat, 28 Feb 2009) | 1 line
give credit where credit is due
........
r70075 | antoine.pitrou | 2009-02-28 13:34:59 -0600 (Sat, 28 Feb 2009) | 3 lines
Amaury's name
........
r70112 | antoine.pitrou | 2009-03-02 17:11:55 -0600 (Mon, 02 Mar 2009) | 4 lines
Looks like this is necessary in order to build cleanly under Windows
(someone correct this if it's wrong, I'm no Windows user)
........
r70133 | benjamin.peterson | 2009-03-03 15:23:32 -0600 (Tue, 03 Mar 2009) | 1 line
fix test_newline_property on _pyio.StringIO
........
r70135 | benjamin.peterson | 2009-03-03 15:47:30 -0600 (Tue, 03 Mar 2009) | 1 line
fix typos and inconsistencies. thanks to Daniel Diniz
........
r70140 | benjamin.peterson | 2009-03-03 16:21:10 -0600 (Tue, 03 Mar 2009) | 1 line
add the test from #5266
........
Added:
python/branches/py3k/Lib/_pyio.py
- copied, changed from r69916, /python/branches/io-c/Lib/_pyio.py
python/branches/py3k/Modules/_bufferedio.c
- copied, changed from r68756, /python/branches/io-c/Modules/_bufferedio.c
python/branches/py3k/Modules/_iobase.c
- copied, changed from r68756, /python/branches/io-c/Modules/_iobase.c
python/branches/py3k/Modules/_iomodule.h
- copied, changed from r68756, /python/branches/io-c/Modules/_iomodule.h
python/branches/py3k/Modules/_textio.c
- copied, changed from r68756, /python/branches/io-c/Modules/_textio.c
python/branches/py3k/Modules/io.c
- copied, changed from r68756, /python/branches/io-c/Modules/io.c
Modified:
python/branches/py3k/ (props changed)
python/branches/py3k/Doc/library/io.rst
python/branches/py3k/Lib/importlib/__init__.py
python/branches/py3k/Lib/importlib/_bootstrap.py
python/branches/py3k/Lib/io.py
python/branches/py3k/Lib/test/test_bufio.py
python/branches/py3k/Lib/test/test_descr.py
python/branches/py3k/Lib/test/test_file.py
python/branches/py3k/Lib/test/test_fileio.py
python/branches/py3k/Lib/test/test_io.py
python/branches/py3k/Lib/test/test_largefile.py
python/branches/py3k/Lib/test/test_memoryio.py
python/branches/py3k/Lib/test/test_univnewlines.py
python/branches/py3k/Lib/test/test_uu.py
python/branches/py3k/Makefile.pre.in
python/branches/py3k/Misc/NEWS
python/branches/py3k/Modules/Setup.dist
python/branches/py3k/Modules/_bytesio.c
python/branches/py3k/Modules/_fileio.c
python/branches/py3k/Modules/_stringio.c
python/branches/py3k/PC/VC6/pythoncore.dsp
python/branches/py3k/PC/config.c
python/branches/py3k/PCbuild/pythoncore.vcproj
python/branches/py3k/Python/pythonrun.c
python/branches/py3k/setup.py
Modified: python/branches/py3k/Doc/library/io.rst
==============================================================================
--- python/branches/py3k/Doc/library/io.rst (original)
+++ python/branches/py3k/Doc/library/io.rst Wed Mar 4 01:14:51 2009
@@ -6,6 +6,8 @@
.. moduleauthor:: Guido van Rossum <guido at python.org>
.. moduleauthor:: Mike Verdone <mike.verdone at gmail.com>
.. moduleauthor:: Mark Russell <mark.russell at zen.co.uk>
+.. moduleauthor:: Antoine Pitrou <solipsis at pitrou.net>
+.. moduleauthor:: Amaury Forgeot d'Arc <amauryfa at gmail.com>
.. sectionauthor:: Benjamin Peterson <benjamin at python.org>
The :mod:`io` module provides the Python interfaces to stream handling. The
@@ -364,6 +366,11 @@
A :exc:`BlockingIOError` is raised if the underlying raw stream has no
data at the moment.
+ .. method:: read1([n])
+
+ Read and return up to *n* bytes, with at most one call to the underlying
+ raw stream's :meth:`~RawIOBase.read` method.
+
.. method:: readinto(b)
Read up to len(b) bytes into bytearray *b* and return the number of bytes
@@ -501,7 +508,7 @@
The constructor creates a :class:`BufferedWriter` for the given writeable
*raw* stream. If the *buffer_size* is not given, it defaults to
- :data:`DEAFULT_BUFFER_SIZE`. If *max_buffer_size* is omitted, it defaults to
+ :data:`DEFAULT_BUFFER_SIZE`. If *max_buffer_size* is omitted, it defaults to
twice the buffer size.
:class:`BufferedWriter` provides or overrides these methods in addition to
Copied: python/branches/py3k/Lib/_pyio.py (from r69916, /python/branches/io-c/Lib/_pyio.py)
==============================================================================
--- /python/branches/io-c/Lib/_pyio.py (original)
+++ python/branches/py3k/Lib/_pyio.py Wed Mar 4 01:14:51 2009
@@ -5,18 +5,22 @@
import os
import abc
import codecs
-#import _fileio
# Import _thread instead of threading to reduce startup cost
try:
from _thread import allocate_lock as Lock
except ImportError:
from _dummy_thread import allocate_lock as Lock
+import io
from io import __all__
# open() uses st_blksize whenever we can
DEFAULT_BUFFER_SIZE = 8 * 1024 # bytes
+# NOTE: Base classes defined here are registered with the "official" ABCs
+# defined in io.py. We don't use real inheritance though, because we don't
+# want to inherit the C implementations.
+
class BlockingIOError(IOError):
@@ -259,7 +263,7 @@
stream.
IOBase also supports the :keyword:`with` statement. In this example,
- fp is closed after the suite of the with statment is complete:
+ fp is closed after the suite of the with statement is complete:
with open('spam.txt', 'r') as fp:
fp.write('Spam and eggs!')
@@ -496,6 +500,8 @@
for line in lines:
self.write(line)
+io.IOBase.register(IOBase)
+
class RawIOBase(IOBase):
@@ -551,7 +557,7 @@
"""
self._unsupported("write")
-
+io.RawIOBase.register(RawIOBase)
from _io import FileIO
RawIOBase.register(FileIO)
@@ -593,6 +599,10 @@
"""
self._unsupported("read")
+ def read1(self, n: int=None) -> bytes:
+ """Read up to n bytes with at most one read() system call."""
+ self._unsupported("read1")
+
def readinto(self, b: bytearray) -> int:
"""Read up to len(b) bytes into b.
@@ -627,6 +637,8 @@
"""
self._unsupported("write")
+io.BufferedIOBase.register(BufferedIOBase)
+
class _BufferedIOMixin(BufferedIOBase):
@@ -908,7 +920,7 @@
def _peek_unlocked(self, n=0):
want = min(n, self.buffer_size)
have = len(self._read_buf) - self._read_pos
- if have < want:
+ if have < want or have <= 0:
to_read = self.buffer_size - have
current = self.raw.read(to_read)
if current:
@@ -921,7 +933,7 @@
# Returns up to n bytes. If at least one byte is buffered, we
# only return buffered bytes. Otherwise, we do one raw read.
if n < 0:
- raise ValueError("number of bytes ot read must be positive")
+ raise ValueError("number of bytes to read must be positive")
if n == 0:
return b""
with self._read_lock:
@@ -930,7 +942,7 @@
min(n, len(self._read_buf) - self._read_pos))
def tell(self):
- return super().tell() - len(self._read_buf) + self._read_pos
+ return _BufferedIOMixin.tell(self) - len(self._read_buf) + self._read_pos
def seek(self, pos, whence=0):
if not (0 <= whence <= 2):
@@ -938,7 +950,7 @@
with self._read_lock:
if whence == 1:
pos -= len(self._read_buf) - self._read_pos
- pos = super().seek(pos, whence)
+ pos = _BufferedIOMixin.seek(self, pos, whence)
self._reset_read_buf()
return pos
@@ -948,7 +960,7 @@
The constructor creates a BufferedWriter for the given writeable raw
stream. If the buffer_size is not given, it defaults to
- DEAFULT_BUFFER_SIZE. If max_buffer_size is omitted, it defaults to
+ DEFAULT_BUFFER_SIZE. If max_buffer_size is omitted, it defaults to
twice the buffer size.
"""
@@ -1026,14 +1038,14 @@
raise BlockingIOError(e.errno, e.strerror, written)
def tell(self):
- return super().tell() + len(self._write_buf)
+ return _BufferedIOMixin.tell(self) + len(self._write_buf)
def seek(self, pos, whence=0):
if not (0 <= whence <= 2):
raise ValueError("invalid whence")
with self._write_lock:
self._flush_unlocked()
- return super().seek(pos, whence)
+ return _BufferedIOMixin.seek(self, pos, whence)
class BufferedRWPair(BufferedIOBase):
@@ -1122,18 +1134,22 @@
if not (0 <= whence <= 2):
raise ValueError("invalid whence")
self.flush()
+ if self._read_buf:
+ # Undo read ahead.
+ with self._read_lock:
+ self.raw.seek(self._read_pos - len(self._read_buf), 1)
# First do the raw seek, then empty the read buffer, so that
# if the raw seek fails, we don't lose buffered data forever.
pos = self.raw.seek(pos, whence)
- if pos < 0:
- raise IOError("seek() returned invalid position")
with self._read_lock:
self._reset_read_buf()
+ if pos < 0:
+ raise IOError("seek() returned invalid position")
return pos
def tell(self):
if self._write_buf:
- return super().tell() + len(self._write_buf)
+ return BufferedWriter.tell(self)
else:
return BufferedReader.tell(self)
@@ -1218,6 +1234,8 @@
"""
return None
+io.TextIOBase.register(TextIOBase)
+
class IncrementalNewlineDecoder(codecs.IncrementalDecoder):
r"""Codec used when reading a file in universal newlines mode. It wraps
@@ -1328,7 +1346,7 @@
write contains a newline character.
"""
- _CHUNK_SIZE = 128
+ _CHUNK_SIZE = 2048
def __init__(self, buffer, encoding=None, errors=None, newline=None,
line_buffering=False):
@@ -1707,7 +1725,9 @@
line = self._get_decoded_chars()
start = 0
- decoder = self._decoder or self._get_decoder()
+ # Make the decoder if it doesn't already exist.
+ if not self._decoder:
+ self._get_decoder()
pos = endpos = None
while True:
@@ -1762,8 +1782,7 @@
endpos = limit # reached length limit
break
- # No line ending seen yet - get more data
- more_line = ''
+ # No line ending seen yet - get more data'
while self._read_chunk():
if self._decoded_chars:
break
Modified: python/branches/py3k/Lib/importlib/__init__.py
==============================================================================
--- python/branches/py3k/Lib/importlib/__init__.py (original)
+++ python/branches/py3k/Lib/importlib/__init__.py Wed Mar 4 01:14:51 2009
@@ -93,12 +93,12 @@
except ImportError:
raise ImportError('posix, nt, or os2 module required for importlib')
_bootstrap._os = _os
-import imp, sys, marshal, errno, _fileio
+import imp, sys, marshal, errno, _io
_bootstrap.imp = imp
_bootstrap.sys = sys
_bootstrap.marshal = marshal
_bootstrap.errno = errno
-_bootstrap._fileio = _fileio
+_bootstrap._io = _io
import _warnings
_bootstrap._warnings = _warnings
Modified: python/branches/py3k/Lib/importlib/_bootstrap.py
==============================================================================
--- python/branches/py3k/Lib/importlib/_bootstrap.py (original)
+++ python/branches/py3k/Lib/importlib/_bootstrap.py Wed Mar 4 01:14:51 2009
@@ -473,7 +473,7 @@
if source_path is None:
return None
import tokenize
- with closing(_fileio._FileIO(source_path, 'r')) as file:
+ with closing(_io.FileIO(source_path, 'r')) as file:
encoding, lines = tokenize.detect_encoding(file.readline)
# XXX Will fail when passed to compile() if the encoding is
# anything other than UTF-8.
@@ -527,7 +527,7 @@
bytecode_path = self.bytecode_path(name)
if not bytecode_path:
bytecode_path = self._base_path + suffix_list(imp.PY_COMPILED)[0]
- file = _fileio._FileIO(bytecode_path, 'w')
+ file = _io.FileIO(bytecode_path, 'w')
try:
with closing(file) as bytecode_file:
bytecode_file.write(data)
Modified: python/branches/py3k/Lib/io.py
==============================================================================
--- python/branches/py3k/Lib/io.py (original)
+++ python/branches/py3k/Lib/io.py Wed Mar 4 01:14:51 2009
@@ -34,9 +34,6 @@
"""
# New I/O library conforming to PEP 3116.
-# This is a prototype; hopefully eventually some of this will be
-# reimplemented in C.
-
# XXX edge cases when switching between reading/writing
# XXX need to support 1 meaning line-buffered
# XXX whenever an argument is None, use the default value
@@ -48,2091 +45,58 @@
__author__ = ("Guido van Rossum <guido at python.org>, "
"Mike Verdone <mike.verdone at gmail.com>, "
- "Mark Russell <mark.russell at zen.co.uk>")
+ "Mark Russell <mark.russell at zen.co.uk>, "
+ "Antoine Pitrou <solipsis at pitrou.net>, "
+ "Amaury Forgeotdarc <amauryfa at gmail.com>")
__all__ = ["BlockingIOError", "open", "IOBase", "RawIOBase", "FileIO",
"BytesIO", "StringIO", "BufferedIOBase",
"BufferedReader", "BufferedWriter", "BufferedRWPair",
"BufferedRandom", "TextIOBase", "TextIOWrapper"]
-import os
-import abc
-import codecs
-import _fileio
-# Import _thread instead of threading to reduce startup cost
-try:
- from _thread import allocate_lock as Lock
-except ImportError:
- from _dummy_thread import allocate_lock as Lock
+import _io
+import abc
# open() uses st_blksize whenever we can
-DEFAULT_BUFFER_SIZE = 8 * 1024 # bytes
-
-
-class BlockingIOError(IOError):
-
- """Exception raised when I/O would block on a non-blocking I/O stream."""
-
- def __init__(self, errno, strerror, characters_written=0):
- IOError.__init__(self, errno, strerror)
- self.characters_written = characters_written
-
-
-def open(file, mode="r", buffering=None, encoding=None, errors=None,
- newline=None, closefd=True):
-
- r"""Open file and return a stream. Raise IOError upon failure.
-
- file is either a text or byte string giving the name (and the path
- if the file isn't in the current working directory) of the file to
- be opened or an integer file descriptor of the file to be
- wrapped. (If a file descriptor is given, it is closed when the
- returned I/O object is closed, unless closefd is set to False.)
-
- mode is an optional string that specifies the mode in which the file
- is opened. It defaults to 'r' which means open for reading in text
- mode. Other common values are 'w' for writing (truncating the file if
- it already exists), and 'a' for appending (which on some Unix systems,
- means that all writes append to the end of the file regardless of the
- current seek position). In text mode, if encoding is not specified the
- encoding used is platform dependent. (For reading and writing raw
- bytes use binary mode and leave encoding unspecified.) The available
- modes are:
-
- ========= ===============================================================
- Character Meaning
- --------- ---------------------------------------------------------------
- 'r' open for reading (default)
- 'w' open for writing, truncating the file first
- 'a' open for writing, appending to the end of the file if it exists
- 'b' binary mode
- 't' text mode (default)
- '+' open a disk file for updating (reading and writing)
- 'U' universal newline mode (for backwards compatibility; unneeded
- for new code)
- ========= ===============================================================
-
- The default mode is 'rt' (open for reading text). For binary random
- access, the mode 'w+b' opens and truncates the file to 0 bytes, while
- 'r+b' opens the file without truncation.
-
- Python distinguishes between files opened in binary and text modes,
- even when the underlying operating system doesn't. Files opened in
- binary mode (appending 'b' to the mode argument) return contents as
- bytes objects without any decoding. In text mode (the default, or when
- 't' is appended to the mode argument), the contents of the file are
- returned as strings, the bytes having been first decoded using a
- platform-dependent encoding or using the specified encoding if given.
-
- buffering is an optional integer used to set the buffering policy. By
- default full buffering is on. Pass 0 to switch buffering off (only
- allowed in binary mode), 1 to set line buffering, and an integer > 1
- for full buffering.
-
- encoding is the name of the encoding used to decode or encode the
- file. This should only be used in text mode. The default encoding is
- platform dependent, but any encoding supported by Python can be
- passed. See the codecs module for the list of supported encodings.
-
- errors is an optional string that specifies how encoding errors are to
- be handled---this argument should not be used in binary mode. Pass
- 'strict' to raise a ValueError exception if there is an encoding error
- (the default of None has the same effect), or pass 'ignore' to ignore
- errors. (Note that ignoring encoding errors can lead to data loss.)
- See the documentation for codecs.register for a list of the permitted
- encoding error strings.
-
- newline controls how universal newlines works (it only applies to text
- mode). It can be None, '', '\n', '\r', and '\r\n'. It works as
- follows:
-
- * On input, if newline is None, universal newlines mode is
- enabled. Lines in the input can end in '\n', '\r', or '\r\n', and
- these are translated into '\n' before being returned to the
- caller. If it is '', universal newline mode is enabled, but line
- endings are returned to the caller untranslated. If it has any of
- the other legal values, input lines are only terminated by the given
- string, and the line ending is returned to the caller untranslated.
-
- * On output, if newline is None, any '\n' characters written are
- translated to the system default line separator, os.linesep. If
- newline is '', no translation takes place. If newline is any of the
- other legal values, any '\n' characters written are translated to
- the given string.
-
- If closefd is False, the underlying file descriptor will be kept open
- when the file is closed. This does not work when a file name is given
- and must be True in that case.
-
- open() returns a file object whose type depends on the mode, and
- through which the standard file operations such as reading and writing
- are performed. When open() is used to open a file in a text mode ('w',
- 'r', 'wt', 'rt', etc.), it returns a TextIOWrapper. When used to open
- a file in a binary mode, the returned class varies: in read binary
- mode, it returns a BufferedReader; in write binary and append binary
- modes, it returns a BufferedWriter, and in read/write mode, it returns
- a BufferedRandom.
-
- It is also possible to use a string or bytearray as a file for both
- reading and writing. For strings StringIO can be used like a file
- opened in a text mode, and for bytes a BytesIO can be used like a file
- opened in a binary mode.
- """
- if not isinstance(file, (str, bytes, int)):
- raise TypeError("invalid file: %r" % file)
- if not isinstance(mode, str):
- raise TypeError("invalid mode: %r" % mode)
- if buffering is not None and not isinstance(buffering, int):
- raise TypeError("invalid buffering: %r" % buffering)
- if encoding is not None and not isinstance(encoding, str):
- raise TypeError("invalid encoding: %r" % encoding)
- if errors is not None and not isinstance(errors, str):
- raise TypeError("invalid errors: %r" % errors)
- modes = set(mode)
- if modes - set("arwb+tU") or len(mode) > len(modes):
- raise ValueError("invalid mode: %r" % mode)
- reading = "r" in modes
- writing = "w" in modes
- appending = "a" in modes
- updating = "+" in modes
- text = "t" in modes
- binary = "b" in modes
- if "U" in modes:
- if writing or appending:
- raise ValueError("can't use U and writing mode at once")
- reading = True
- if text and binary:
- raise ValueError("can't have text and binary mode at once")
- if reading + writing + appending > 1:
- raise ValueError("can't have read/write/append mode at once")
- if not (reading or writing or appending):
- raise ValueError("must have exactly one of read/write/append mode")
- if binary and encoding is not None:
- raise ValueError("binary mode doesn't take an encoding argument")
- if binary and errors is not None:
- raise ValueError("binary mode doesn't take an errors argument")
- if binary and newline is not None:
- raise ValueError("binary mode doesn't take a newline argument")
- raw = FileIO(file,
- (reading and "r" or "") +
- (writing and "w" or "") +
- (appending and "a" or "") +
- (updating and "+" or ""),
- closefd)
- if buffering is None:
- buffering = -1
- line_buffering = False
- if buffering == 1 or buffering < 0 and raw.isatty():
- buffering = -1
- line_buffering = True
- if buffering < 0:
- buffering = DEFAULT_BUFFER_SIZE
- try:
- bs = os.fstat(raw.fileno()).st_blksize
- except (os.error, AttributeError):
- pass
- else:
- if bs > 1:
- buffering = bs
- if buffering < 0:
- raise ValueError("invalid buffering size")
- if buffering == 0:
- if binary:
- return raw
- raise ValueError("can't have unbuffered text I/O")
- if updating:
- buffer = BufferedRandom(raw, buffering)
- elif writing or appending:
- buffer = BufferedWriter(raw, buffering)
- elif reading:
- buffer = BufferedReader(raw, buffering)
- else:
- raise ValueError("unknown mode: %r" % mode)
- if binary:
- return buffer
- text = TextIOWrapper(buffer, encoding, errors, newline, line_buffering)
- text.mode = mode
- return text
-
-class _DocDescriptor:
- """Helper for builtins.open.__doc__
- """
- def __get__(self, obj, typ):
- return (
- "open(file, mode='r', buffering=None, encoding=None, "
- "errors=None, newline=None, closefd=True)\n\n" +
- open.__doc__)
-
-class OpenWrapper:
- """Wrapper for builtins.open
-
- Trick so that open won't become a bound method when stored
- as a class variable (as dbm.dumb does).
-
- See initstdio() in Python/pythonrun.c.
- """
- __doc__ = _DocDescriptor()
-
- def __new__(cls, *args, **kwargs):
- return open(*args, **kwargs)
-
-
-class UnsupportedOperation(ValueError, IOError):
+DEFAULT_BUFFER_SIZE = _io.DEFAULT_BUFFER_SIZE
+BlockingIOError = _io.BlockingIOError
+UnsupportedOperation = _io.UnsupportedOperation
+open = _io.open
+OpenWrapper = _io.open
+
+# Declaring ABCs in C is tricky so we do it here.
+# Method descriptions and default implementations are inherited from the C
+# version however.
+class IOBase(_io._IOBase, metaclass=abc.ABCMeta):
pass
+class RawIOBase(_io._RawIOBase, IOBase):
+ pass
-class IOBase(metaclass=abc.ABCMeta):
-
- """The abstract base class for all I/O classes, acting on streams of
- bytes. There is no public constructor.
-
- This class provides dummy implementations for many methods that
- derived classes can override selectively; the default implementations
- represent a file that cannot be read, written or seeked.
-
- Even though IOBase does not declare read, readinto, or write because
- their signatures will vary, implementations and clients should
- consider those methods part of the interface. Also, implementations
- may raise a IOError when operations they do not support are called.
-
- The basic type used for binary data read from or written to a file is
- bytes. bytearrays are accepted too, and in some cases (such as
- readinto) needed. Text I/O classes work with str data.
-
- Note that calling any method (even inquiries) on a closed stream is
- undefined. Implementations may raise IOError in this case.
-
- IOBase (and its subclasses) support the iterator protocol, meaning
- that an IOBase object can be iterated over yielding the lines in a
- stream.
-
- IOBase also supports the :keyword:`with` statement. In this example,
- fp is closed after the suite of the with statment is complete:
-
- with open('spam.txt', 'r') as fp:
- fp.write('Spam and eggs!')
- """
-
- ### Internal ###
-
- def _unsupported(self, name: str) -> IOError:
- """Internal: raise an exception for unsupported operations."""
- raise UnsupportedOperation("%s.%s() not supported" %
- (self.__class__.__name__, name))
-
- ### Positioning ###
-
- def seek(self, pos: int, whence: int = 0) -> int:
- """Change stream position.
-
- Change the stream position to byte offset offset. offset is
- interpreted relative to the position indicated by whence. Values
- for whence are:
-
- * 0 -- start of stream (the default); offset should be zero or positive
- * 1 -- current stream position; offset may be negative
- * 2 -- end of stream; offset is usually negative
-
- Return the new absolute position.
- """
- self._unsupported("seek")
-
- def tell(self) -> int:
- """Return current stream position."""
- self._checkClosed()
- return self.seek(0, 1)
-
- def truncate(self, pos: int = None) -> int:
- """Truncate file to size bytes.
-
- Size defaults to the current IO position as reported by tell(). Return
- the new size.
- """
- self._unsupported("truncate")
-
- ### Flush and close ###
-
- def flush(self) -> None:
- """Flush write buffers, if applicable.
-
- This is not implemented for read-only and non-blocking streams.
- """
- # XXX Should this return the number of bytes written???
- if self.__closed:
- raise ValueError("I/O operation on closed file.")
-
- __closed = False
-
- def close(self) -> None:
- """Flush and close the IO object.
-
- This method has no effect if the file is already closed.
- """
- if not self.__closed:
- try:
- self.flush()
- except IOError:
- pass # If flush() fails, just give up
- self.__closed = True
-
- def __del__(self) -> None:
- """Destructor. Calls close()."""
- # The try/except block is in case this is called at program
- # exit time, when it's possible that globals have already been
- # deleted, and then the close() call might fail. Since
- # there's nothing we can do about such failures and they annoy
- # the end users, we suppress the traceback.
- try:
- self.close()
- except:
- pass
-
- ### Inquiries ###
-
- def seekable(self) -> bool:
- """Return whether object supports random access.
-
- If False, seek(), tell() and truncate() will raise IOError.
- This method may need to do a test seek().
- """
- return False
-
- def _checkSeekable(self, msg=None):
- """Internal: raise an IOError if file is not seekable
- """
- if not self.seekable():
- raise IOError("File or stream is not seekable."
- if msg is None else msg)
-
-
- def readable(self) -> bool:
- """Return whether object was opened for reading.
-
- If False, read() will raise IOError.
- """
- return False
-
- def _checkReadable(self, msg=None):
- """Internal: raise an IOError if file is not readable
- """
- if not self.readable():
- raise IOError("File or stream is not readable."
- if msg is None else msg)
-
- def writable(self) -> bool:
- """Return whether object was opened for writing.
-
- If False, write() and truncate() will raise IOError.
- """
- return False
-
- def _checkWritable(self, msg=None):
- """Internal: raise an IOError if file is not writable
- """
- if not self.writable():
- raise IOError("File or stream is not writable."
- if msg is None else msg)
-
- @property
- def closed(self):
- """closed: bool. True iff the file has been closed.
-
- For backwards compatibility, this is a property, not a predicate.
- """
- return self.__closed
-
- def _checkClosed(self, msg=None):
- """Internal: raise an ValueError if file is closed
- """
- if self.closed:
- raise ValueError("I/O operation on closed file."
- if msg is None else msg)
-
- ### Context manager ###
-
- def __enter__(self) -> "IOBase": # That's a forward reference
- """Context management protocol. Returns self."""
- self._checkClosed()
- return self
-
- def __exit__(self, *args) -> None:
- """Context management protocol. Calls close()"""
- self.close()
-
- ### Lower-level APIs ###
-
- # XXX Should these be present even if unimplemented?
-
- def fileno(self) -> int:
- """Returns underlying file descriptor if one exists.
-
- An IOError is raised if the IO object does not use a file descriptor.
- """
- self._unsupported("fileno")
-
- def isatty(self) -> bool:
- """Return whether this is an 'interactive' stream.
-
- Return False if it can't be determined.
- """
- self._checkClosed()
- return False
-
- ### Readline[s] and writelines ###
-
- def readline(self, limit: int = -1) -> bytes:
- r"""Read and return a line from the stream.
-
- If limit is specified, at most limit bytes will be read.
-
- The line terminator is always b'\n' for binary files; for text
- files, the newlines argument to open can be used to select the line
- terminator(s) recognized.
- """
- # For backwards compatibility, a (slowish) readline().
- if hasattr(self, "peek"):
- def nreadahead():
- readahead = self.peek(1)
- if not readahead:
- return 1
- n = (readahead.find(b"\n") + 1) or len(readahead)
- if limit >= 0:
- n = min(n, limit)
- return n
- else:
- def nreadahead():
- return 1
- if limit is None:
- limit = -1
- res = bytearray()
- while limit < 0 or len(res) < limit:
- b = self.read(nreadahead())
- if not b:
- break
- res += b
- if res.endswith(b"\n"):
- break
- return bytes(res)
-
- def __iter__(self):
- self._checkClosed()
- return self
-
- def __next__(self):
- line = self.readline()
- if not line:
- raise StopIteration
- return line
-
- def readlines(self, hint=None):
- """Return a list of lines from the stream.
-
- hint can be specified to control the number of lines read: no more
- lines will be read if the total size (in bytes/characters) of all
- lines so far exceeds hint.
- """
- if hint is None or hint <= 0:
- return list(self)
- n = 0
- lines = []
- for line in self:
- lines.append(line)
- n += len(line)
- if n >= hint:
- break
- return lines
-
- def writelines(self, lines):
- self._checkClosed()
- for line in lines:
- self.write(line)
-
-
-class RawIOBase(IOBase):
-
- """Base class for raw binary I/O."""
-
- # The read() method is implemented by calling readinto(); derived
- # classes that want to support read() only need to implement
- # readinto() as a primitive operation. In general, readinto() can be
- # more efficient than read().
-
- # (It would be tempting to also provide an implementation of
- # readinto() in terms of read(), in case the latter is a more suitable
- # primitive operation, but that would lead to nasty recursion in case
- # a subclass doesn't implement either.)
-
- def read(self, n: int = -1) -> bytes:
- """Read and return up to n bytes.
-
- Returns an empty bytes object on EOF, or None if the object is
- set not to block and has no data to read.
- """
- self._checkClosed()
- if n is None:
- n = -1
- if n < 0:
- return self.readall()
- b = bytearray(n.__index__())
- n = self.readinto(b)
- del b[n:]
- return bytes(b)
-
- def readall(self):
- """Read until EOF, using multiple read() call."""
- self._checkClosed()
- res = bytearray()
- while True:
- data = self.read(DEFAULT_BUFFER_SIZE)
- if not data:
- break
- res += data
- return bytes(res)
-
- def readinto(self, b: bytearray) -> int:
- """Read up to len(b) bytes into b.
-
- Returns number of bytes read (0 for EOF), or None if the object
- is set not to block as has no data to read.
- """
- self._unsupported("readinto")
-
- def write(self, b: bytes) -> int:
- """Write the given buffer to the IO stream.
-
- Returns the number of bytes written, which may be less than len(b).
- """
- self._unsupported("write")
-
-
-class FileIO(_fileio._FileIO, RawIOBase):
-
- """Raw I/O implementation for OS files."""
-
- # This multiply inherits from _FileIO and RawIOBase to make
- # isinstance(io.FileIO(), io.RawIOBase) return True without requiring
- # that _fileio._FileIO inherits from io.RawIOBase (which would be hard
- # to do since _fileio.c is written in C).
-
- def __init__(self, name, mode="r", closefd=True):
- _fileio._FileIO.__init__(self, name, mode, closefd)
- self._name = name
-
- def close(self):
- _fileio._FileIO.close(self)
- RawIOBase.close(self)
-
- @property
- def name(self):
- return self._name
-
-
-class BufferedIOBase(IOBase):
-
- """Base class for buffered IO objects.
-
- The main difference with RawIOBase is that the read() method
- supports omitting the size argument, and does not have a default
- implementation that defers to readinto().
-
- In addition, read(), readinto() and write() may raise
- BlockingIOError if the underlying raw stream is in non-blocking
- mode and not ready; unlike their raw counterparts, they will never
- return None.
-
- A typical implementation should not inherit from a RawIOBase
- implementation, but wrap one.
- """
-
- def read(self, n: int = None) -> bytes:
- """Read and return up to n bytes.
-
- If the argument is omitted, None, or negative, reads and
- returns all data until EOF.
-
- If the argument is positive, and the underlying raw stream is
- not 'interactive', multiple raw reads may be issued to satisfy
- the byte count (unless EOF is reached first). But for
- interactive raw streams (XXX and for pipes?), at most one raw
- read will be issued, and a short result does not imply that
- EOF is imminent.
-
- Returns an empty bytes array on EOF.
-
- Raises BlockingIOError if the underlying raw stream has no
- data at the moment.
- """
- self._unsupported("read")
-
- def readinto(self, b: bytearray) -> int:
- """Read up to len(b) bytes into b.
-
- Like read(), this may issue multiple reads to the underlying raw
- stream, unless the latter is 'interactive'.
-
- Returns the number of bytes read (0 for EOF).
-
- Raises BlockingIOError if the underlying raw stream has no
- data at the moment.
- """
- # XXX This ought to work with anything that supports the buffer API
- self._checkClosed()
- data = self.read(len(b))
- n = len(data)
- try:
- b[:n] = data
- except TypeError as err:
- import array
- if not isinstance(b, array.array):
- raise err
- b[:n] = array.array('b', data)
- return n
-
- def write(self, b: bytes) -> int:
- """Write the given buffer to the IO stream.
-
- Return the number of bytes written, which is never less than
- len(b).
-
- Raises BlockingIOError if the buffer is full and the
- underlying raw stream cannot accept more data at the moment.
- """
- self._unsupported("write")
-
-
-class _BufferedIOMixin(BufferedIOBase):
-
- """A mixin implementation of BufferedIOBase with an underlying raw stream.
-
- This passes most requests on to the underlying raw stream. It
- does *not* provide implementations of read(), readinto() or
- write().
- """
-
- def __init__(self, raw):
- self.raw = raw
-
- ### Positioning ###
-
- def seek(self, pos, whence=0):
- return self.raw.seek(pos, whence)
-
- def tell(self):
- return self.raw.tell()
-
- def truncate(self, pos=None):
- # Flush the stream. We're mixing buffered I/O with lower-level I/O,
- # and a flush may be necessary to synch both views of the current
- # file state.
- self.flush()
-
- if pos is None:
- pos = self.tell()
- # XXX: Should seek() be used, instead of passing the position
- # XXX directly to truncate?
- return self.raw.truncate(pos)
-
- ### Flush and close ###
-
- def flush(self):
- self.raw.flush()
-
- def close(self):
- if not self.closed:
- try:
- self.flush()
- except IOError:
- pass # If flush() fails, just give up
- self.raw.close()
-
- ### Inquiries ###
-
- def seekable(self):
- return self.raw.seekable()
-
- def readable(self):
- return self.raw.readable()
-
- def writable(self):
- return self.raw.writable()
-
- @property
- def closed(self):
- return self.raw.closed
-
- @property
- def name(self):
- return self.raw.name
-
- @property
- def mode(self):
- return self.raw.mode
-
- ### Lower-level APIs ###
-
- def fileno(self):
- return self.raw.fileno()
-
- def isatty(self):
- return self.raw.isatty()
-
-
-class _BytesIO(BufferedIOBase):
-
- """Buffered I/O implementation using an in-memory bytes buffer."""
-
- def __init__(self, initial_bytes=None):
- buf = bytearray()
- if initial_bytes is not None:
- buf += initial_bytes
- self._buffer = buf
- self._pos = 0
-
- def getvalue(self):
- """Return the bytes value (contents) of the buffer
- """
- self._checkClosed()
- return bytes(self._buffer)
-
- def read(self, n=None):
- self._checkClosed()
- if n is None:
- n = -1
- if n < 0:
- n = len(self._buffer)
- if len(self._buffer) <= self._pos:
- return b""
- newpos = min(len(self._buffer), self._pos + n)
- b = self._buffer[self._pos : newpos]
- self._pos = newpos
- return bytes(b)
-
- def read1(self, n):
- """This is the same as read.
- """
- return self.read(n)
-
- def write(self, b):
- self._checkClosed()
- if isinstance(b, str):
- raise TypeError("can't write str to binary stream")
- n = len(b)
- if n == 0:
- return 0
- pos = self._pos
- if pos > len(self._buffer):
- # Inserts null bytes between the current end of the file
- # and the new write position.
- padding = b'\x00' * (pos - len(self._buffer))
- self._buffer += padding
- self._buffer[pos:pos + n] = b
- self._pos += n
- return n
-
- def seek(self, pos, whence=0):
- self._checkClosed()
- try:
- pos = pos.__index__()
- except AttributeError as err:
- raise TypeError("an integer is required") from err
- if whence == 0:
- if pos < 0:
- raise ValueError("negative seek position %r" % (pos,))
- self._pos = pos
- elif whence == 1:
- self._pos = max(0, self._pos + pos)
- elif whence == 2:
- self._pos = max(0, len(self._buffer) + pos)
- else:
- raise ValueError("invalid whence value")
- return self._pos
-
- def tell(self):
- self._checkClosed()
- return self._pos
-
- def truncate(self, pos=None):
- self._checkClosed()
- if pos is None:
- pos = self._pos
- elif pos < 0:
- raise ValueError("negative truncate position %r" % (pos,))
- del self._buffer[pos:]
- return self.seek(pos)
-
- def readable(self):
- return True
-
- def writable(self):
- return True
-
- def seekable(self):
- return True
-
-# Use the faster implementation of BytesIO if available
-try:
- import _bytesio
-
- class BytesIO(_bytesio._BytesIO, BufferedIOBase):
- __doc__ = _bytesio._BytesIO.__doc__
-
-except ImportError:
- BytesIO = _BytesIO
-
-
-class BufferedReader(_BufferedIOMixin):
-
- """BufferedReader(raw[, buffer_size])
-
- A buffer for a readable, sequential BaseRawIO object.
-
- The constructor creates a BufferedReader for the given readable raw
- stream and buffer_size. If buffer_size is omitted, DEFAULT_BUFFER_SIZE
- is used.
- """
-
- def __init__(self, raw, buffer_size=DEFAULT_BUFFER_SIZE):
- """Create a new buffered reader using the given readable raw IO object.
- """
- raw._checkReadable()
- _BufferedIOMixin.__init__(self, raw)
- self.buffer_size = buffer_size
- self._reset_read_buf()
- self._read_lock = Lock()
-
- def _reset_read_buf(self):
- self._read_buf = b""
- self._read_pos = 0
-
- def read(self, n=None):
- """Read n bytes.
-
- Returns exactly n bytes of data unless the underlying raw IO
- stream reaches EOF or if the call would block in non-blocking
- mode. If n is negative, read until EOF or until read() would
- block.
- """
- self._checkClosed()
- with self._read_lock:
- return self._read_unlocked(n)
-
- def _read_unlocked(self, n=None):
- nodata_val = b""
- empty_values = (b"", None)
- buf = self._read_buf
- pos = self._read_pos
-
- # Special case for when the number of bytes to read is unspecified.
- if n is None or n == -1:
- self._reset_read_buf()
- chunks = [buf[pos:]] # Strip the consumed bytes.
- current_size = 0
- while True:
- # Read until EOF or until read() would block.
- chunk = self.raw.read()
- if chunk in empty_values:
- nodata_val = chunk
- break
- current_size += len(chunk)
- chunks.append(chunk)
- return b"".join(chunks) or nodata_val
-
- # The number of bytes to read is specified, return at most n bytes.
- avail = len(buf) - pos # Length of the available buffered data.
- if n <= avail:
- # Fast path: the data to read is fully buffered.
- self._read_pos += n
- return buf[pos:pos+n]
- # Slow path: read from the stream until enough bytes are read,
- # or until an EOF occurs or until read() would block.
- chunks = [buf[pos:]]
- wanted = max(self.buffer_size, n)
- while avail < n:
- chunk = self.raw.read(wanted)
- if chunk in empty_values:
- nodata_val = chunk
- break
- avail += len(chunk)
- chunks.append(chunk)
- # n is more then avail only when an EOF occurred or when
- # read() would have blocked.
- n = min(n, avail)
- out = b"".join(chunks)
- self._read_buf = out[n:] # Save the extra data in the buffer.
- self._read_pos = 0
- return out[:n] if out else nodata_val
-
- def peek(self, n=0):
- """Returns buffered bytes without advancing the position.
-
- The argument indicates a desired minimal number of bytes; we
- do at most one raw read to satisfy it. We never return more
- than self.buffer_size.
- """
- self._checkClosed()
- with self._read_lock:
- return self._peek_unlocked(n)
-
- def _peek_unlocked(self, n=0):
- want = min(n, self.buffer_size)
- have = len(self._read_buf) - self._read_pos
- if have < want:
- to_read = self.buffer_size - have
- current = self.raw.read(to_read)
- if current:
- self._read_buf = self._read_buf[self._read_pos:] + current
- self._read_pos = 0
- return self._read_buf[self._read_pos:]
-
- def read1(self, n):
- """Reads up to n bytes, with at most one read() system call."""
- # Returns up to n bytes. If at least one byte is buffered, we
- # only return buffered bytes. Otherwise, we do one raw read.
- self._checkClosed()
- if n <= 0:
- return b""
- with self._read_lock:
- self._peek_unlocked(1)
- return self._read_unlocked(
- min(n, len(self._read_buf) - self._read_pos))
-
- def tell(self):
- self._checkClosed()
- return self.raw.tell() - len(self._read_buf) + self._read_pos
-
- def seek(self, pos, whence=0):
- self._checkClosed()
- with self._read_lock:
- if whence == 1:
- pos -= len(self._read_buf) - self._read_pos
- pos = self.raw.seek(pos, whence)
- self._reset_read_buf()
- return pos
-
-
-class BufferedWriter(_BufferedIOMixin):
-
- """A buffer for a writeable sequential RawIO object.
-
- The constructor creates a BufferedWriter for the given writeable raw
- stream. If the buffer_size is not given, it defaults to
- DEAFULT_BUFFER_SIZE. If max_buffer_size is omitted, it defaults to
- twice the buffer size.
- """
-
- def __init__(self, raw,
- buffer_size=DEFAULT_BUFFER_SIZE, max_buffer_size=None):
- raw._checkWritable()
- _BufferedIOMixin.__init__(self, raw)
- self.buffer_size = buffer_size
- self.max_buffer_size = (2*buffer_size
- if max_buffer_size is None
- else max_buffer_size)
- self._write_buf = bytearray()
- self._write_lock = Lock()
-
- def write(self, b):
- self._checkClosed()
- if isinstance(b, str):
- raise TypeError("can't write str to binary stream")
- with self._write_lock:
- # XXX we can implement some more tricks to try and avoid
- # partial writes
- if len(self._write_buf) > self.buffer_size:
- # We're full, so let's pre-flush the buffer
- try:
- self._flush_unlocked()
- except BlockingIOError as e:
- # We can't accept anything else.
- # XXX Why not just let the exception pass through?
- raise BlockingIOError(e.errno, e.strerror, 0)
- before = len(self._write_buf)
- self._write_buf.extend(b)
- written = len(self._write_buf) - before
- if len(self._write_buf) > self.buffer_size:
- try:
- self._flush_unlocked()
- except BlockingIOError as e:
- if len(self._write_buf) > self.max_buffer_size:
- # We've hit max_buffer_size. We have to accept a
- # partial write and cut back our buffer.
- overage = len(self._write_buf) - self.max_buffer_size
- self._write_buf = self._write_buf[:self.max_buffer_size]
- raise BlockingIOError(e.errno, e.strerror, overage)
- return written
-
- def truncate(self, pos=None):
- self._checkClosed()
- with self._write_lock:
- self._flush_unlocked()
- if pos is None:
- pos = self.raw.tell()
- return self.raw.truncate(pos)
-
- def flush(self):
- self._checkClosed()
- with self._write_lock:
- self._flush_unlocked()
-
- def _flush_unlocked(self):
- written = 0
- try:
- while self._write_buf:
- n = self.raw.write(self._write_buf)
- del self._write_buf[:n]
- written += n
- except BlockingIOError as e:
- n = e.characters_written
- del self._write_buf[:n]
- written += n
- raise BlockingIOError(e.errno, e.strerror, written)
-
- def tell(self):
- self._checkClosed()
- return self.raw.tell() + len(self._write_buf)
-
- def seek(self, pos, whence=0):
- self._checkClosed()
- with self._write_lock:
- self._flush_unlocked()
- return self.raw.seek(pos, whence)
-
-
-class BufferedRWPair(BufferedIOBase):
-
- """A buffered reader and writer object together.
-
- A buffered reader object and buffered writer object put together to
- form a sequential IO object that can read and write. This is typically
- used with a socket or two-way pipe.
-
- reader and writer are RawIOBase objects that are readable and
- writeable respectively. If the buffer_size is omitted it defaults to
- DEFAULT_BUFFER_SIZE. The max_buffer_size (for the buffered writer)
- defaults to twice the buffer size.
- """
-
- # XXX The usefulness of this (compared to having two separate IO
- # objects) is questionable.
-
- def __init__(self, reader, writer,
- buffer_size=DEFAULT_BUFFER_SIZE, max_buffer_size=None):
- """Constructor.
-
- The arguments are two RawIO instances.
- """
- reader._checkReadable()
- writer._checkWritable()
- self.reader = BufferedReader(reader, buffer_size)
- self.writer = BufferedWriter(writer, buffer_size, max_buffer_size)
-
- def read(self, n=None):
- if n is None:
- n = -1
- return self.reader.read(n)
-
- def readinto(self, b):
- return self.reader.readinto(b)
-
- def write(self, b):
- return self.writer.write(b)
-
- def peek(self, n=0):
- return self.reader.peek(n)
-
- def read1(self, n):
- return self.reader.read1(n)
-
- def readable(self):
- return self.reader.readable()
-
- def writable(self):
- return self.writer.writable()
-
- def flush(self):
- return self.writer.flush()
-
- def close(self):
- self.writer.close()
- self.reader.close()
-
- def isatty(self):
- return self.reader.isatty() or self.writer.isatty()
-
- @property
- def closed(self):
- return self.writer.closed
-
-
-class BufferedRandom(BufferedWriter, BufferedReader):
-
- """A buffered interface to random access streams.
-
- The constructor creates a reader and writer for a seekable stream,
- raw, given in the first argument. If the buffer_size is omitted it
- defaults to DEFAULT_BUFFER_SIZE. The max_buffer_size (for the buffered
- writer) defaults to twice the buffer size.
- """
-
- def __init__(self, raw,
- buffer_size=DEFAULT_BUFFER_SIZE, max_buffer_size=None):
- raw._checkSeekable()
- BufferedReader.__init__(self, raw, buffer_size)
- BufferedWriter.__init__(self, raw, buffer_size, max_buffer_size)
-
- def seek(self, pos, whence=0):
- self.flush()
- # First do the raw seek, then empty the read buffer, so that
- # if the raw seek fails, we don't lose buffered data forever.
- pos = self.raw.seek(pos, whence)
- with self._read_lock:
- self._reset_read_buf()
- return pos
-
- def tell(self):
- self._checkClosed()
- if self._write_buf:
- return self.raw.tell() + len(self._write_buf)
- else:
- return BufferedReader.tell(self)
-
- def truncate(self, pos=None):
- if pos is None:
- pos = self.tell()
- # Use seek to flush the read buffer.
- self.seek(pos)
- return BufferedWriter.truncate(self)
-
- def read(self, n=None):
- if n is None:
- n = -1
- self.flush()
- return BufferedReader.read(self, n)
-
- def readinto(self, b):
- self.flush()
- return BufferedReader.readinto(self, b)
-
- def peek(self, n=0):
- self.flush()
- return BufferedReader.peek(self, n)
-
- def read1(self, n):
- self.flush()
- return BufferedReader.read1(self, n)
-
- def write(self, b):
- self._checkClosed()
- if self._read_buf:
- # Undo readahead
- with self._read_lock:
- self.raw.seek(self._read_pos - len(self._read_buf), 1)
- self._reset_read_buf()
- return BufferedWriter.write(self, b)
-
-
-class TextIOBase(IOBase):
-
- """Base class for text I/O.
-
- This class provides a character and line based interface to stream
- I/O. There is no readinto method because Python's character strings
- are immutable. There is no public constructor.
- """
-
- def read(self, n: int = -1) -> str:
- """Read at most n characters from stream.
-
- Read from underlying buffer until we have n characters or we hit EOF.
- If n is negative or omitted, read until EOF.
- """
- self._unsupported("read")
-
- def write(self, s: str) -> int:
- """Write string s to stream."""
- self._unsupported("write")
-
- def truncate(self, pos: int = None) -> int:
- """Truncate size to pos."""
- self._unsupported("truncate")
-
- def readline(self) -> str:
- """Read until newline or EOF.
-
- Returns an empty string if EOF is hit immediately.
- """
- self._unsupported("readline")
-
- @property
- def encoding(self):
- """Subclasses should override."""
- return None
-
- @property
- def newlines(self):
- """Line endings translated so far.
-
- Only line endings translated during reading are considered.
-
- Subclasses should override.
- """
- return None
-
-
-class IncrementalNewlineDecoder(codecs.IncrementalDecoder):
- r"""Codec used when reading a file in universal newlines mode. It wraps
- another incremental decoder, translating \r\n and \r into \n. It also
- records the types of newlines encountered. When used with
- translate=False, it ensures that the newline sequence is returned in
- one piece.
- """
- def __init__(self, decoder, translate, errors='strict'):
- codecs.IncrementalDecoder.__init__(self, errors=errors)
- self.translate = translate
- self.decoder = decoder
- self.seennl = 0
- self.pendingcr = False
-
- def decode(self, input, final=False):
- # decode input (with the eventual \r from a previous pass)
- output = self.decoder.decode(input, final=final)
- if self.pendingcr and (output or final):
- output = "\r" + output
- self.pendingcr = False
-
- # retain last \r even when not translating data:
- # then readline() is sure to get \r\n in one pass
- if output.endswith("\r") and not final:
- output = output[:-1]
- self.pendingcr = True
-
- # Record which newlines are read
- crlf = output.count('\r\n')
- cr = output.count('\r') - crlf
- lf = output.count('\n') - crlf
- self.seennl |= (lf and self._LF) | (cr and self._CR) \
- | (crlf and self._CRLF)
-
- if self.translate:
- if crlf:
- output = output.replace("\r\n", "\n")
- if cr:
- output = output.replace("\r", "\n")
-
- return output
-
- def getstate(self):
- buf, flag = self.decoder.getstate()
- flag <<= 1
- if self.pendingcr:
- flag |= 1
- return buf, flag
-
- def setstate(self, state):
- buf, flag = state
- self.pendingcr = bool(flag & 1)
- self.decoder.setstate((buf, flag >> 1))
-
- def reset(self):
- self.seennl = 0
- self.pendingcr = False
- self.decoder.reset()
-
- _LF = 1
- _CR = 2
- _CRLF = 4
-
- @property
- def newlines(self):
- return (None,
- "\n",
- "\r",
- ("\r", "\n"),
- "\r\n",
- ("\n", "\r\n"),
- ("\r", "\r\n"),
- ("\r", "\n", "\r\n")
- )[self.seennl]
-
-
-class TextIOWrapper(TextIOBase):
-
- r"""Character and line based layer over a BufferedIOBase object, buffer.
-
- encoding gives the name of the encoding that the stream will be
- decoded or encoded with. It defaults to locale.getpreferredencoding.
-
- errors determines the strictness of encoding and decoding (see the
- codecs.register) and defaults to "strict".
-
- newline can be None, '', '\n', '\r', or '\r\n'. It controls the
- handling of line endings. If it is None, universal newlines is
- enabled. With this enabled, on input, the lines endings '\n', '\r',
- or '\r\n' are translated to '\n' before being returned to the
- caller. Conversely, on output, '\n' is translated to the system
- default line separator, os.linesep. If newline is any other of its
- legal values, that newline becomes the newline when the file is read
- and it is returned untranslated. On output, '\n' is converted to the
- newline.
-
- If line_buffering is True, a call to flush is implied when a call to
- write contains a newline character.
- """
-
- _CHUNK_SIZE = 2048
-
- def __init__(self, buffer, encoding=None, errors=None, newline=None,
- line_buffering=False):
- if newline not in (None, "", "\n", "\r", "\r\n"):
- raise ValueError("illegal newline value: %r" % (newline,))
- if encoding is None:
- try:
- encoding = os.device_encoding(buffer.fileno())
- except (AttributeError, UnsupportedOperation):
- pass
- if encoding is None:
- try:
- import locale
- except ImportError:
- # Importing locale may fail if Python is being built
- encoding = "ascii"
- else:
- encoding = locale.getpreferredencoding()
-
- if not isinstance(encoding, str):
- raise ValueError("invalid encoding: %r" % encoding)
-
- if errors is None:
- errors = "strict"
- else:
- if not isinstance(errors, str):
- raise ValueError("invalid errors: %r" % errors)
-
- self.buffer = buffer
- self._line_buffering = line_buffering
- self._encoding = encoding
- self._errors = errors
- self._readuniversal = not newline
- self._readtranslate = newline is None
- self._readnl = newline
- self._writetranslate = newline != ''
- self._writenl = newline or os.linesep
- self._encoder = None
- self._decoder = None
- self._decoded_chars = '' # buffer for text returned from decoder
- self._decoded_chars_used = 0 # offset into _decoded_chars for read()
- self._snapshot = None # info for reconstructing decoder state
- self._seekable = self._telling = self.buffer.seekable()
-
- # self._snapshot is either None, or a tuple (dec_flags, next_input)
- # where dec_flags is the second (integer) item of the decoder state
- # and next_input is the chunk of input bytes that comes next after the
- # snapshot point. We use this to reconstruct decoder states in tell().
-
- # Naming convention:
- # - "bytes_..." for integer variables that count input bytes
- # - "chars_..." for integer variables that count decoded characters
-
- @property
- def encoding(self):
- return self._encoding
-
- @property
- def errors(self):
- return self._errors
-
- @property
- def line_buffering(self):
- return self._line_buffering
-
- def seekable(self):
- return self._seekable
-
- def readable(self):
- return self.buffer.readable()
-
- def writable(self):
- return self.buffer.writable()
-
- def flush(self):
- self.buffer.flush()
- self._telling = self._seekable
-
- def close(self):
- try:
- self.flush()
- except:
- pass # If flush() fails, just give up
- self.buffer.close()
-
- @property
- def closed(self):
- return self.buffer.closed
-
- @property
- def name(self):
- return self.buffer.name
-
- def fileno(self):
- return self.buffer.fileno()
-
- def isatty(self):
- return self.buffer.isatty()
-
- def write(self, s: str):
- self._checkClosed()
- if not isinstance(s, str):
- raise TypeError("can't write %s to text stream" %
- s.__class__.__name__)
- length = len(s)
- haslf = (self._writetranslate or self._line_buffering) and "\n" in s
- if haslf and self._writetranslate and self._writenl != "\n":
- s = s.replace("\n", self._writenl)
- encoder = self._encoder or self._get_encoder()
- # XXX What if we were just reading?
- b = encoder.encode(s)
- self.buffer.write(b)
- if self._line_buffering and (haslf or "\r" in s):
- self.flush()
- self._snapshot = None
- if self._decoder:
- self._decoder.reset()
- return length
-
- def _get_encoder(self):
- make_encoder = codecs.getincrementalencoder(self._encoding)
- self._encoder = make_encoder(self._errors)
- return self._encoder
-
- def _get_decoder(self):
- make_decoder = codecs.getincrementaldecoder(self._encoding)
- decoder = make_decoder(self._errors)
- if self._readuniversal:
- decoder = IncrementalNewlineDecoder(decoder, self._readtranslate)
- self._decoder = decoder
- return decoder
-
- # The following three methods implement an ADT for _decoded_chars.
- # Text returned from the decoder is buffered here until the client
- # requests it by calling our read() or readline() method.
- def _set_decoded_chars(self, chars):
- """Set the _decoded_chars buffer."""
- self._decoded_chars = chars
- self._decoded_chars_used = 0
-
- def _get_decoded_chars(self, n=None):
- """Advance into the _decoded_chars buffer."""
- offset = self._decoded_chars_used
- if n is None:
- chars = self._decoded_chars[offset:]
- else:
- chars = self._decoded_chars[offset:offset + n]
- self._decoded_chars_used += len(chars)
- return chars
-
- def _rewind_decoded_chars(self, n):
- """Rewind the _decoded_chars buffer."""
- if self._decoded_chars_used < n:
- raise AssertionError("rewind decoded_chars out of bounds")
- self._decoded_chars_used -= n
-
- def _read_chunk(self):
- """
- Read and decode the next chunk of data from the BufferedReader.
- """
-
- # The return value is True unless EOF was reached. The decoded
- # string is placed in self._decoded_chars (replacing its previous
- # value). The entire input chunk is sent to the decoder, though
- # some of it may remain buffered in the decoder, yet to be
- # converted.
-
- if self._decoder is None:
- raise ValueError("no decoder")
-
- if self._telling:
- # To prepare for tell(), we need to snapshot a point in the
- # file where the decoder's input buffer is empty.
-
- dec_buffer, dec_flags = self._decoder.getstate()
- # Given this, we know there was a valid snapshot point
- # len(dec_buffer) bytes ago with decoder state (b'', dec_flags).
-
- # Read a chunk, decode it, and put the result in self._decoded_chars.
- input_chunk = self.buffer.read1(self._CHUNK_SIZE)
- eof = not input_chunk
- self._set_decoded_chars(self._decoder.decode(input_chunk, eof))
-
- if self._telling:
- # At the snapshot point, len(dec_buffer) bytes before the read,
- # the next input to be decoded is dec_buffer + input_chunk.
- self._snapshot = (dec_flags, dec_buffer + input_chunk)
-
- return not eof
-
- def _pack_cookie(self, position, dec_flags=0,
- bytes_to_feed=0, need_eof=0, chars_to_skip=0):
- # The meaning of a tell() cookie is: seek to position, set the
- # decoder flags to dec_flags, read bytes_to_feed bytes, feed them
- # into the decoder with need_eof as the EOF flag, then skip
- # chars_to_skip characters of the decoded result. For most simple
- # decoders, tell() will often just give a byte offset in the file.
- return (position | (dec_flags<<64) | (bytes_to_feed<<128) |
- (chars_to_skip<<192) | bool(need_eof)<<256)
-
- def _unpack_cookie(self, bigint):
- rest, position = divmod(bigint, 1<<64)
- rest, dec_flags = divmod(rest, 1<<64)
- rest, bytes_to_feed = divmod(rest, 1<<64)
- need_eof, chars_to_skip = divmod(rest, 1<<64)
- return position, dec_flags, bytes_to_feed, need_eof, chars_to_skip
-
- def tell(self):
- self._checkClosed()
- if not self._seekable:
- raise IOError("underlying stream is not seekable")
- if not self._telling:
- raise IOError("telling position disabled by next() call")
- self.flush()
- position = self.buffer.tell()
- decoder = self._decoder
- if decoder is None or self._snapshot is None:
- if self._decoded_chars:
- # This should never happen.
- raise AssertionError("pending decoded text")
- return position
-
- # Skip backward to the snapshot point (see _read_chunk).
- dec_flags, next_input = self._snapshot
- position -= len(next_input)
-
- # How many decoded characters have been used up since the snapshot?
- chars_to_skip = self._decoded_chars_used
- if chars_to_skip == 0:
- # We haven't moved from the snapshot point.
- return self._pack_cookie(position, dec_flags)
-
- # Starting from the snapshot position, we will walk the decoder
- # forward until it gives us enough decoded characters.
- saved_state = decoder.getstate()
- try:
- # Note our initial start point.
- decoder.setstate((b'', dec_flags))
- start_pos = position
- start_flags, bytes_fed, chars_decoded = dec_flags, 0, 0
- need_eof = 0
-
- # Feed the decoder one byte at a time. As we go, note the
- # nearest "safe start point" before the current location
- # (a point where the decoder has nothing buffered, so seek()
- # can safely start from there and advance to this location).
- next_byte = bytearray(1)
- for next_byte[0] in next_input:
- bytes_fed += 1
- chars_decoded += len(decoder.decode(next_byte))
- dec_buffer, dec_flags = decoder.getstate()
- if not dec_buffer and chars_decoded <= chars_to_skip:
- # Decoder buffer is empty, so this is a safe start point.
- start_pos += bytes_fed
- chars_to_skip -= chars_decoded
- start_flags, bytes_fed, chars_decoded = dec_flags, 0, 0
- if chars_decoded >= chars_to_skip:
- break
- else:
- # We didn't get enough decoded data; signal EOF to get more.
- chars_decoded += len(decoder.decode(b'', final=True))
- need_eof = 1
- if chars_decoded < chars_to_skip:
- raise IOError("can't reconstruct logical file position")
-
- # The returned cookie corresponds to the last safe start point.
- return self._pack_cookie(
- start_pos, start_flags, bytes_fed, need_eof, chars_to_skip)
- finally:
- decoder.setstate(saved_state)
-
- def truncate(self, pos=None):
- self.flush()
- if pos is None:
- pos = self.tell()
- self.seek(pos)
- return self.buffer.truncate()
-
- def seek(self, cookie, whence=0):
- self._checkClosed()
- if not self._seekable:
- raise IOError("underlying stream is not seekable")
- if whence == 1: # seek relative to current position
- if cookie != 0:
- raise IOError("can't do nonzero cur-relative seeks")
- # Seeking to the current position should attempt to
- # sync the underlying buffer with the current position.
- whence = 0
- cookie = self.tell()
- if whence == 2: # seek relative to end of file
- if cookie != 0:
- raise IOError("can't do nonzero end-relative seeks")
- self.flush()
- position = self.buffer.seek(0, 2)
- self._set_decoded_chars('')
- self._snapshot = None
- if self._decoder:
- self._decoder.reset()
- return position
- if whence != 0:
- raise ValueError("invalid whence (%r, should be 0, 1 or 2)" %
- (whence,))
- if cookie < 0:
- raise ValueError("negative seek position %r" % (cookie,))
- self.flush()
-
- # The strategy of seek() is to go back to the safe start point
- # and replay the effect of read(chars_to_skip) from there.
- start_pos, dec_flags, bytes_to_feed, need_eof, chars_to_skip = \
- self._unpack_cookie(cookie)
-
- # Seek back to the safe start point.
- self.buffer.seek(start_pos)
- self._set_decoded_chars('')
- self._snapshot = None
-
- # Restore the decoder to its state from the safe start point.
- if self._decoder or dec_flags or chars_to_skip:
- self._decoder = self._decoder or self._get_decoder()
- self._decoder.setstate((b'', dec_flags))
- self._snapshot = (dec_flags, b'')
-
- if chars_to_skip:
- # Just like _read_chunk, feed the decoder and save a snapshot.
- input_chunk = self.buffer.read(bytes_to_feed)
- self._set_decoded_chars(
- self._decoder.decode(input_chunk, need_eof))
- self._snapshot = (dec_flags, input_chunk)
-
- # Skip chars_to_skip of the decoded characters.
- if len(self._decoded_chars) < chars_to_skip:
- raise IOError("can't restore logical file position")
- self._decoded_chars_used = chars_to_skip
-
- return cookie
-
- def read(self, n=None):
- self._checkClosed()
- if n is None:
- n = -1
- decoder = self._decoder or self._get_decoder()
- if n < 0:
- # Read everything.
- result = (self._get_decoded_chars() +
- decoder.decode(self.buffer.read(), final=True))
- self._set_decoded_chars('')
- self._snapshot = None
- return result
- else:
- # Keep reading chunks until we have n characters to return.
- eof = False
- result = self._get_decoded_chars(n)
- while len(result) < n and not eof:
- eof = not self._read_chunk()
- result += self._get_decoded_chars(n - len(result))
- return result
-
- def __next__(self):
- self._checkClosed()
- self._telling = False
- line = self.readline()
- if not line:
- self._snapshot = None
- self._telling = self._seekable
- raise StopIteration
- return line
-
- def readline(self, limit=None):
- self._checkClosed()
- if limit is None:
- limit = -1
-
- # Grab all the decoded text (we will rewind any extra bits later).
- line = self._get_decoded_chars()
-
- start = 0
- decoder = self._decoder or self._get_decoder()
-
- pos = endpos = None
- while True:
- if self._readtranslate:
- # Newlines are already translated, only search for \n
- pos = line.find('\n', start)
- if pos >= 0:
- endpos = pos + 1
- break
- else:
- start = len(line)
-
- elif self._readuniversal:
- # Universal newline search. Find any of \r, \r\n, \n
- # The decoder ensures that \r\n are not split in two pieces
-
- # In C we'd look for these in parallel of course.
- nlpos = line.find("\n", start)
- crpos = line.find("\r", start)
- if crpos == -1:
- if nlpos == -1:
- # Nothing found
- start = len(line)
- else:
- # Found \n
- endpos = nlpos + 1
- break
- elif nlpos == -1:
- # Found lone \r
- endpos = crpos + 1
- break
- elif nlpos < crpos:
- # Found \n
- endpos = nlpos + 1
- break
- elif nlpos == crpos + 1:
- # Found \r\n
- endpos = crpos + 2
- break
- else:
- # Found \r
- endpos = crpos + 1
- break
- else:
- # non-universal
- pos = line.find(self._readnl)
- if pos >= 0:
- endpos = pos + len(self._readnl)
- break
-
- if limit >= 0 and len(line) >= limit:
- endpos = limit # reached length limit
- break
-
- # No line ending seen yet - get more data
- more_line = ''
- while self._read_chunk():
- if self._decoded_chars:
- break
- if self._decoded_chars:
- line += self._get_decoded_chars()
- else:
- # end of file
- self._set_decoded_chars('')
- self._snapshot = None
- return line
-
- if limit >= 0 and endpos > limit:
- endpos = limit # don't exceed limit
-
- # Rewind _decoded_chars to just after the line ending we found.
- self._rewind_decoded_chars(len(line) - endpos)
- return line[:endpos]
-
- @property
- def newlines(self):
- return self._decoder.newlines if self._decoder else None
-
-class _StringIO(TextIOWrapper):
- """Text I/O implementation using an in-memory buffer.
-
- The initial_value argument sets the value of object. The newline
- argument is like the one of TextIOWrapper's constructor.
- """
-
- # XXX This is really slow, but fully functional
-
- def __init__(self, initial_value="", newline="\n"):
- super(_StringIO, self).__init__(BytesIO(),
- encoding="utf-8",
- errors="strict",
- newline=newline)
- if initial_value:
- if not isinstance(initial_value, str):
- initial_value = str(initial_value)
- self.write(initial_value)
- self.seek(0)
-
- def getvalue(self):
- self.flush()
- return self.buffer.getvalue().decode(self._encoding, self._errors)
-
-try:
- import _stringio
-
- # This subclass is a reimplementation of the TextIOWrapper
- # interface without any of its text decoding facilities. All the
- # stored data is manipulated with the efficient
- # _stringio._StringIO extension type. Also, the newline decoding
- # mechanism of IncrementalNewlineDecoder is reimplemented here for
- # efficiency. Doing otherwise, would require us to implement a
- # fake decoder which would add an additional and unnecessary layer
- # on top of the _StringIO methods.
-
- class StringIO(_stringio._StringIO, TextIOBase):
- """Text I/O implementation using an in-memory buffer.
-
- The initial_value argument sets the value of object. The newline
- argument is like the one of TextIOWrapper's constructor.
- """
-
- _CHUNK_SIZE = 4096
-
- def __init__(self, initial_value="", newline="\n"):
- if newline not in (None, "", "\n", "\r", "\r\n"):
- raise ValueError("illegal newline value: %r" % (newline,))
-
- self._readuniversal = not newline
- self._readtranslate = newline is None
- self._readnl = newline
- self._writetranslate = newline != ""
- self._writenl = newline or os.linesep
- self._pending = ""
- self._seennl = 0
-
- # Reset the buffer first, in case __init__ is called
- # multiple times.
- self.truncate(0)
- if initial_value is None:
- initial_value = ""
- self.write(initial_value)
- self.seek(0)
-
- @property
- def buffer(self):
- raise UnsupportedOperation("%s.buffer attribute is unsupported" %
- self.__class__.__name__)
-
- # XXX Cruft to support the TextIOWrapper API. This would only
- # be meaningful if StringIO supported the buffer attribute.
- # Hopefully, a better solution, than adding these pseudo-attributes,
- # will be found.
- @property
- def encoding(self):
- return "utf-8"
-
- @property
- def errors(self):
- return "strict"
-
- @property
- def line_buffering(self):
- return False
-
- def _decode_newlines(self, input, final=False):
- # decode input (with the eventual \r from a previous pass)
- if self._pending:
- input = self._pending + input
-
- # retain last \r even when not translating data:
- # then readline() is sure to get \r\n in one pass
- if input.endswith("\r") and not final:
- input = input[:-1]
- self._pending = "\r"
- else:
- self._pending = ""
-
- # Record which newlines are read
- crlf = input.count('\r\n')
- cr = input.count('\r') - crlf
- lf = input.count('\n') - crlf
- self._seennl |= (lf and self._LF) | (cr and self._CR) \
- | (crlf and self._CRLF)
-
- if self._readtranslate:
- if crlf:
- output = input.replace("\r\n", "\n")
- if cr:
- output = input.replace("\r", "\n")
- else:
- output = input
-
- return output
-
- def writable(self):
- return True
-
- def readable(self):
- return True
-
- def seekable(self):
- return True
-
- _read = _stringio._StringIO.read
- _write = _stringio._StringIO.write
- _tell = _stringio._StringIO.tell
- _seek = _stringio._StringIO.seek
- _truncate = _stringio._StringIO.truncate
- _getvalue = _stringio._StringIO.getvalue
-
- def getvalue(self) -> str:
- """Retrieve the entire contents of the object."""
- self._checkClosed()
- return self._getvalue()
-
- def write(self, s: str) -> int:
- """Write string s to file.
-
- Returns the number of characters written.
- """
- self._checkClosed()
- if not isinstance(s, str):
- raise TypeError("can't write %s to text stream" %
- s.__class__.__name__)
- length = len(s)
- if self._writetranslate and self._writenl != "\n":
- s = s.replace("\n", self._writenl)
- self._pending = ""
- self._write(s)
- return length
-
- def read(self, n: int = None) -> str:
- """Read at most n characters, returned as a string.
-
- If the argument is negative or omitted, read until EOF
- is reached. Return an empty string at EOF.
- """
- self._checkClosed()
- if n is None:
- n = -1
- res = self._pending
- if n < 0:
- res += self._decode_newlines(self._read(), True)
- self._pending = ""
- return res
- else:
- res = self._decode_newlines(self._read(n), True)
- self._pending = res[n:]
- return res[:n]
-
- def tell(self) -> int:
- """Tell the current file position."""
- self._checkClosed()
- if self._pending:
- return self._tell() - len(self._pending)
- else:
- return self._tell()
-
- def seek(self, pos: int = None, whence: int = 0) -> int:
- """Change stream position.
-
- Seek to character offset pos relative to position indicated by whence:
- 0 Start of stream (the default). pos should be >= 0;
- 1 Current position - pos must be 0;
- 2 End of stream - pos must be 0.
- Returns the new absolute position.
- """
- self._checkClosed()
- self._pending = ""
- return self._seek(pos, whence)
-
- def truncate(self, pos: int = None) -> int:
- """Truncate size to pos.
-
- The pos argument defaults to the current file position, as
- returned by tell(). Imply an absolute seek to pos.
- Returns the new absolute position.
- """
- self._checkClosed()
- self._pending = ""
- return self._truncate(pos)
-
- def readline(self, limit: int = None) -> str:
- self._checkClosed()
- if limit is None:
- limit = -1
- if limit >= 0:
- # XXX: Hack to support limit argument, for backwards
- # XXX compatibility
- line = self.readline()
- if len(line) <= limit:
- return line
- line, self._pending = line[:limit], line[limit:] + self._pending
- return line
-
- line = self._pending
- self._pending = ""
-
- start = 0
- pos = endpos = None
- while True:
- if self._readtranslate:
- # Newlines are already translated, only search for \n
- pos = line.find('\n', start)
- if pos >= 0:
- endpos = pos + 1
- break
- else:
- start = len(line)
-
- elif self._readuniversal:
- # Universal newline search. Find any of \r, \r\n, \n
- # The decoder ensures that \r\n are not split in two pieces
-
- # In C we'd look for these in parallel of course.
- nlpos = line.find("\n", start)
- crpos = line.find("\r", start)
- if crpos == -1:
- if nlpos == -1:
- # Nothing found
- start = len(line)
- else:
- # Found \n
- endpos = nlpos + 1
- break
- elif nlpos == -1:
- # Found lone \r
- endpos = crpos + 1
- break
- elif nlpos < crpos:
- # Found \n
- endpos = nlpos + 1
- break
- elif nlpos == crpos + 1:
- # Found \r\n
- endpos = crpos + 2
- break
- else:
- # Found \r
- endpos = crpos + 1
- break
- else:
- # non-universal
- pos = line.find(self._readnl)
- if pos >= 0:
- endpos = pos + len(self._readnl)
- break
-
- # No line ending seen yet - get more data
- more_line = self.read(self._CHUNK_SIZE)
- if more_line:
- line += more_line
- else:
- # end of file
- return line
-
- self._pending = line[endpos:]
- return line[:endpos]
-
- _LF = 1
- _CR = 2
- _CRLF = 4
+class BufferedIOBase(_io._BufferedIOBase, IOBase):
+ pass
- @property
- def newlines(self):
- return (None,
- "\n",
- "\r",
- ("\r", "\n"),
- "\r\n",
- ("\n", "\r\n"),
- ("\r", "\r\n"),
- ("\r", "\n", "\r\n")
- )[self._seennl]
+class TextIOBase(_io._TextIOBase, IOBase):
+ pass
+FileIO = _io.FileIO
+BytesIO = _io.BytesIO
+StringIO = _io.StringIO
+BufferedReader = _io.BufferedReader
+BufferedWriter = _io.BufferedWriter
+BufferedRWPair = _io.BufferedRWPair
+BufferedRandom = _io.BufferedRandom
+IncrementalNewlineDecoder = _io.IncrementalNewlineDecoder
+TextIOWrapper = _io.TextIOWrapper
+
+RawIOBase.register(FileIO)
+
+BufferedIOBase.register(BytesIO)
+BufferedIOBase.register(BufferedReader)
+BufferedIOBase.register(BufferedWriter)
+BufferedIOBase.register(BufferedRandom)
+BufferedIOBase.register(BufferedRWPair)
-except ImportError:
- StringIO = _StringIO
+TextIOBase.register(StringIO)
+TextIOBase.register(TextIOWrapper)
Modified: python/branches/py3k/Lib/test/test_bufio.py
==============================================================================
--- python/branches/py3k/Lib/test/test_bufio.py (original)
+++ python/branches/py3k/Lib/test/test_bufio.py Wed Mar 4 01:14:51 2009
@@ -1,9 +1,12 @@
import unittest
from test import support
-# Simple test to ensure that optimizations in fileobject.c deliver
-# the expected results. For best testing, run this under a debug-build
-# Python too (to exercise asserts in the C code).
+import io # C implementation.
+import _pyio as pyio # Python implementation.
+
+# Simple test to ensure that optimizations in the IO library deliver the
+# expected results. For best testing, run this under a debug-build Python too
+# (to exercise asserts in the C code).
lengths = list(range(1, 257)) + [512, 1000, 1024, 2048, 4096, 8192, 10000,
16384, 32768, 65536, 1000000]
@@ -18,7 +21,7 @@
# Since C doesn't guarantee we can write/read arbitrary bytes in text
# files, use binary mode.
- f = open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
try:
# write once with \n and once without
f.write(s)
@@ -58,8 +61,16 @@
def test_nullpat(self):
self.drive_one(bytes(1000))
+
+class CBufferSizeTest(BufferSizeTest):
+ open = io.open
+
+class PyBufferSizeTest(BufferSizeTest):
+ open = staticmethod(pyio.open)
+
+
def test_main():
- support.run_unittest(BufferSizeTest)
+ support.run_unittest(CBufferSizeTest, PyBufferSizeTest)
if __name__ == "__main__":
test_main()
Modified: python/branches/py3k/Lib/test/test_descr.py
==============================================================================
--- python/branches/py3k/Lib/test/test_descr.py (original)
+++ python/branches/py3k/Lib/test/test_descr.py Wed Mar 4 01:14:51 2009
@@ -2605,10 +2605,10 @@
def test_descrdoc(self):
# Testing descriptor doc strings...
- from _fileio import _FileIO
+ from _io import FileIO
def check(descr, what):
self.assertEqual(descr.__doc__, what)
- check(_FileIO.closed, "True if the file is closed") # getset descriptor
+ check(FileIO.closed, "True if the file is closed") # getset descriptor
check(complex.real, "the real part of a complex number") # member descriptor
def test_doc_descriptor(self):
Modified: python/branches/py3k/Lib/test/test_file.py
==============================================================================
--- python/branches/py3k/Lib/test/test_file.py (original)
+++ python/branches/py3k/Lib/test/test_file.py Wed Mar 4 01:14:51 2009
@@ -4,6 +4,9 @@
from array import array
from weakref import proxy
+import io
+import _pyio as pyio
+
from test.support import TESTFN, findfile, run_unittest
from collections import UserList
@@ -11,7 +14,7 @@
# file tests for which a test file is automatically set up
def setUp(self):
- self.f = open(TESTFN, 'wb')
+ self.f = self.open(TESTFN, 'wb')
def tearDown(self):
if self.f:
@@ -39,7 +42,7 @@
self.f.write(b'12')
self.f.close()
a = array('b', b'x'*10)
- self.f = open(TESTFN, 'rb')
+ self.f = self.open(TESTFN, 'rb')
n = self.f.readinto(a)
self.assertEquals(b'12', a.tostring()[:n])
@@ -47,7 +50,7 @@
# verify readinto refuses text files
a = array('b', b'x'*10)
self.f.close()
- self.f = open(TESTFN, 'r')
+ self.f = self.open(TESTFN, 'r')
if hasattr(self.f, "readinto"):
self.assertRaises(TypeError, self.f.readinto, a)
@@ -56,7 +59,7 @@
l = UserList([b'1', b'2'])
self.f.writelines(l)
self.f.close()
- self.f = open(TESTFN, 'rb')
+ self.f = self.open(TESTFN, 'rb')
buf = self.f.read()
self.assertEquals(buf, b'12')
@@ -126,13 +129,20 @@
def testReadWhenWriting(self):
self.assertRaises(IOError, self.f.read)
+class CAutoFileTests(AutoFileTests):
+ open = io.open
+
+class PyAutoFileTests(AutoFileTests):
+ open = staticmethod(pyio.open)
+
+
class OtherFileTests(unittest.TestCase):
def testModeStrings(self):
# check invalid mode strings
for mode in ("", "aU", "wU+"):
try:
- f = open(TESTFN, mode)
+ f = self.open(TESTFN, mode)
except ValueError:
pass
else:
@@ -153,7 +163,7 @@
# verify that we get a sensible error message for bad mode argument
bad_mode = "qwerty"
try:
- f = open(TESTFN, bad_mode)
+ f = self.open(TESTFN, bad_mode)
except ValueError as msg:
if msg.args[0] != 0:
s = str(msg)
@@ -170,11 +180,11 @@
# misbehaviour especially with repeated close() calls
for s in (-1, 0, 1, 512):
try:
- f = open(TESTFN, 'wb', s)
+ f = self.open(TESTFN, 'wb', s)
f.write(str(s).encode("ascii"))
f.close()
f.close()
- f = open(TESTFN, 'rb', s)
+ f = self.open(TESTFN, 'rb', s)
d = int(f.read().decode("ascii"))
f.close()
f.close()
@@ -187,13 +197,13 @@
# "file.truncate fault on windows"
os.unlink(TESTFN)
- f = open(TESTFN, 'wb')
+ f = self.open(TESTFN, 'wb')
try:
f.write(b'12345678901') # 11 bytes
f.close()
- f = open(TESTFN,'rb+')
+ f = self.open(TESTFN,'rb+')
data = f.read(5)
if data != b'12345':
self.fail("Read on file opened for update failed %r" % data)
@@ -233,13 +243,13 @@
try:
# Prepare the testfile
- bag = open(TESTFN, "wb")
+ bag = self.open(TESTFN, "wb")
bag.write(filler * nchunks)
bag.writelines(testlines)
bag.close()
# Test for appropriate errors mixing read* and iteration
for methodname, args in methods:
- f = open(TESTFN, 'rb')
+ f = self.open(TESTFN, 'rb')
if next(f) != filler:
self.fail, "Broken testfile"
meth = getattr(f, methodname)
@@ -253,7 +263,7 @@
# ("h", "a", "m", "\n"), so 4096 lines of that should get us
# exactly on the buffer boundary for any power-of-2 buffersize
# between 4 and 16384 (inclusive).
- f = open(TESTFN, 'rb')
+ f = self.open(TESTFN, 'rb')
for i in range(nchunks):
next(f)
testline = testlines.pop(0)
@@ -295,7 +305,7 @@
self.fail("readlines() after next() with empty buffer "
"failed. Got %r, expected %r" % (line, testline))
# Reading after iteration hit EOF shouldn't hurt either
- f = open(TESTFN, 'rb')
+ f = self.open(TESTFN, 'rb')
try:
for line in f:
pass
@@ -311,12 +321,19 @@
finally:
os.unlink(TESTFN)
+class COtherFileTests(OtherFileTests):
+ open = io.open
+
+class PyOtherFileTests(OtherFileTests):
+ open = staticmethod(pyio.open)
+
def test_main():
# Historically, these tests have been sloppy about removing TESTFN.
# So get rid of it no matter what.
try:
- run_unittest(AutoFileTests, OtherFileTests)
+ run_unittest(CAutoFileTests, PyAutoFileTests,
+ COtherFileTests, PyOtherFileTests)
finally:
if os.path.exists(TESTFN):
os.unlink(TESTFN)
Modified: python/branches/py3k/Lib/test/test_fileio.py
==============================================================================
--- python/branches/py3k/Lib/test/test_fileio.py (original)
+++ python/branches/py3k/Lib/test/test_fileio.py Wed Mar 4 01:14:51 2009
@@ -10,13 +10,13 @@
make_bad_fd)
from collections import UserList
-import _fileio
+from _io import FileIO as _FileIO
class AutoFileTests(unittest.TestCase):
# file tests for which a test file is automatically set up
def setUp(self):
- self.f = _fileio._FileIO(TESTFN, 'w')
+ self.f = _FileIO(TESTFN, 'w')
def tearDown(self):
if self.f:
@@ -63,13 +63,13 @@
self.f.write(bytes([1, 2]))
self.f.close()
a = array('b', b'x'*10)
- self.f = _fileio._FileIO(TESTFN, 'r')
+ self.f = _FileIO(TESTFN, 'r')
n = self.f.readinto(a)
self.assertEquals(array('b', [1, 2]), a[:n])
def testRepr(self):
self.assertEquals(repr(self.f),
- "_fileio._FileIO(%d, %s)" % (self.f.fileno(),
+ "io.FileIO(%d, %s)" % (self.f.fileno(),
repr(self.f.mode)))
def testErrors(self):
@@ -80,7 +80,7 @@
self.assertRaises(ValueError, f.read, 10) # Open for reading
f.close()
self.assert_(f.closed)
- f = _fileio._FileIO(TESTFN, 'r')
+ f = _FileIO(TESTFN, 'r')
self.assertRaises(TypeError, f.readinto, "")
self.assert_(not f.closed)
f.close()
@@ -106,7 +106,7 @@
# Windows always returns "[Errno 13]: Permission denied
# Unix calls dircheck() and returns "[Errno 21]: Is a directory"
try:
- _fileio._FileIO('.', 'r')
+ _FileIO('.', 'r')
except IOError as e:
self.assertNotEqual(e.errno, 0)
self.assertEqual(e.filename, ".")
@@ -118,19 +118,19 @@
def testAbles(self):
try:
- f = _fileio._FileIO(TESTFN, "w")
+ f = _FileIO(TESTFN, "w")
self.assertEquals(f.readable(), False)
self.assertEquals(f.writable(), True)
self.assertEquals(f.seekable(), True)
f.close()
- f = _fileio._FileIO(TESTFN, "r")
+ f = _FileIO(TESTFN, "r")
self.assertEquals(f.readable(), True)
self.assertEquals(f.writable(), False)
self.assertEquals(f.seekable(), True)
f.close()
- f = _fileio._FileIO(TESTFN, "a+")
+ f = _FileIO(TESTFN, "a+")
self.assertEquals(f.readable(), True)
self.assertEquals(f.writable(), True)
self.assertEquals(f.seekable(), True)
@@ -139,14 +139,14 @@
if sys.platform != "win32":
try:
- f = _fileio._FileIO("/dev/tty", "a")
+ f = _FileIO("/dev/tty", "a")
except EnvironmentError:
# When run in a cron job there just aren't any
# ttys, so skip the test. This also handles other
# OS'es that don't support /dev/tty.
pass
else:
- f = _fileio._FileIO("/dev/tty", "a")
+ f = _FileIO("/dev/tty", "a")
self.assertEquals(f.readable(), False)
self.assertEquals(f.writable(), True)
if sys.platform != "darwin" and \
@@ -163,7 +163,7 @@
# check invalid mode strings
for mode in ("", "aU", "wU+", "rw", "rt"):
try:
- f = _fileio._FileIO(TESTFN, mode)
+ f = _FileIO(TESTFN, mode)
except ValueError:
pass
else:
@@ -172,10 +172,26 @@
def testUnicodeOpen(self):
# verify repr works for unicode too
- f = _fileio._FileIO(str(TESTFN), "w")
+ f = _FileIO(str(TESTFN), "w")
f.close()
os.unlink(TESTFN)
+ def testBytesOpen(self):
+ # Opening a bytes filename
+ try:
+ fn = TESTFN.encode("ascii")
+ except UnicodeEncodeError:
+ # Skip test
+ return
+ f = _FileIO(fn, "w")
+ try:
+ f.write(b"abc")
+ f.close()
+ with open(TESTFN, "rb") as f:
+ self.assertEquals(f.read(), b"abc")
+ finally:
+ os.unlink(TESTFN)
+
def testInvalidFd(self):
self.assertRaises(ValueError, _fileio._FileIO, -10)
self.assertRaises(OSError, _fileio._FileIO, make_bad_fd())
@@ -184,7 +200,7 @@
# verify that we get a sensible error message for bad mode argument
bad_mode = "qwerty"
try:
- f = _fileio._FileIO(TESTFN, bad_mode)
+ f = _FileIO(TESTFN, bad_mode)
except ValueError as msg:
if msg.args[0] != 0:
s = str(msg)
@@ -200,11 +216,11 @@
def bug801631():
# SF bug <http://www.python.org/sf/801631>
# "file.truncate fault on windows"
- f = _fileio._FileIO(TESTFN, 'w')
+ f = _FileIO(TESTFN, 'w')
f.write(bytes(range(11)))
f.close()
- f = _fileio._FileIO(TESTFN,'r+')
+ f = _FileIO(TESTFN,'r+')
data = f.read(5)
if data != bytes(range(5)):
self.fail("Read on file opened for update failed %r" % data)
@@ -244,14 +260,14 @@
pass
def testInvalidInit(self):
- self.assertRaises(TypeError, _fileio._FileIO, "1", 0, 0)
+ self.assertRaises(TypeError, _FileIO, "1", 0, 0)
def testWarnings(self):
with check_warnings() as w:
self.assertEqual(w.warnings, [])
- self.assertRaises(TypeError, _fileio._FileIO, [])
+ self.assertRaises(TypeError, _FileIO, [])
self.assertEqual(w.warnings, [])
- self.assertRaises(ValueError, _fileio._FileIO, "/some/invalid/name", "rt")
+ self.assertRaises(ValueError, _FileIO, "/some/invalid/name", "rt")
self.assertEqual(w.warnings, [])
Modified: python/branches/py3k/Lib/test/test_io.py
==============================================================================
--- python/branches/py3k/Lib/test/test_io.py (original)
+++ python/branches/py3k/Lib/test/test_io.py Wed Mar 4 01:14:51 2009
@@ -1,4 +1,23 @@
-"""Unit tests for io.py."""
+"""Unit tests for the io module."""
+
+# Tests of io are scattered over the test suite:
+# * test_bufio - tests file buffering
+# * test_memoryio - tests BytesIO and StringIO
+# * test_fileio - tests FileIO
+# * test_file - tests the file interface
+# * test_io - tests everything else in the io module
+# * test_univnewlines - tests universal newline support
+# * test_largefile - tests operations on a file greater than 2**32 bytes
+# (only enabled with -ulargefile)
+
+################################################################################
+# ATTENTION TEST WRITERS!!!
+################################################################################
+# When writing tests for io, it's important to test both the C and Python
+# implementations. This is usually done by writing a base test that refers to
+# the type it is testing as a attribute. Then it provides custom subclasses to
+# test both implementations. This file has lots of examples.
+################################################################################
import os
import sys
@@ -7,27 +26,40 @@
import threading
import random
import unittest
-from itertools import chain, cycle
+import weakref
+import gc
+import abc
+from itertools import chain, cycle, count
+from collections import deque
from test import support
import codecs
-import io # The module under test
+import io # C implementation of io
+import _pyio as pyio # Python implementation of io
+
+def _default_chunk_size():
+ """Get the default TextIOWrapper chunk size"""
+ with open(__file__, "r", encoding="latin1") as f:
+ return f._CHUNK_SIZE
-class MockRawIO(io.RawIOBase):
+
+class MockRawIO:
def __init__(self, read_stack=()):
self._read_stack = list(read_stack)
self._write_stack = []
+ self._reads = 0
def read(self, n=None):
+ self._reads += 1
try:
return self._read_stack.pop(0)
except:
return b""
def write(self, b):
- self._write_stack.append(b[:])
+ self._write_stack.append(bytes(b))
return len(b)
def writable(self):
@@ -43,41 +75,148 @@
return True
def seek(self, pos, whence):
- pass
+ return 0 # wrong but we gotta return something
def tell(self):
- return 42
+ return 0 # same comment as above
+
+ def readinto(self, buf):
+ self._reads += 1
+ max_len = len(buf)
+ try:
+ data = self._read_stack[0]
+ except IndexError:
+ return 0
+ if data is None:
+ del self._read_stack[0]
+ return None
+ n = len(data)
+ if len(data) <= max_len:
+ del self._read_stack[0]
+ buf[:n] = data
+ return n
+ else:
+ buf[:] = data[:max_len]
+ self._read_stack[0] = data[max_len:]
+ return max_len
+
+ def truncate(self, pos=None):
+ return pos
+
+class CMockRawIO(MockRawIO, io.RawIOBase):
+ pass
+
+class PyMockRawIO(MockRawIO, pyio.RawIOBase):
+ pass
+
+
+class MisbehavedRawIO(MockRawIO):
+ def write(self, b):
+ return super().write(b) * 2
+
+ def read(self, n=None):
+ return super().read(n) * 2
+
+ def seek(self, pos, whence):
+ return -123
+
+ def tell(self):
+ return -456
+
+ def readinto(self, buf):
+ super().readinto(buf)
+ return len(buf) * 5
+
+class CMisbehavedRawIO(MisbehavedRawIO, io.RawIOBase):
+ pass
+
+class PyMisbehavedRawIO(MisbehavedRawIO, pyio.RawIOBase):
+ pass
+
+
+class CloseFailureIO(MockRawIO):
+ closed = 0
+
+ def close(self):
+ if not self.closed:
+ self.closed = 1
+ raise IOError
+
+class CCloseFailureIO(CloseFailureIO, io.RawIOBase):
+ pass
+class PyCloseFailureIO(CloseFailureIO, pyio.RawIOBase):
+ pass
-class MockFileIO(io.BytesIO):
+
+class MockFileIO:
def __init__(self, data):
self.read_history = []
- io.BytesIO.__init__(self, data)
+ super().__init__(data)
def read(self, n=None):
- res = io.BytesIO.read(self, n)
+ res = super().read(n)
self.read_history.append(None if res is None else len(res))
return res
+ def readinto(self, b):
+ res = super().readinto(b)
+ self.read_history.append(res)
+ return res
+
+class CMockFileIO(MockFileIO, io.BytesIO):
+ pass
+
+class PyMockFileIO(MockFileIO, pyio.BytesIO):
+ pass
-class MockNonBlockWriterIO(io.RawIOBase):
- def __init__(self, blocking_script):
- self._blocking_script = list(blocking_script)
+class MockNonBlockWriterIO:
+
+ def __init__(self):
self._write_stack = []
+ self._blocker_char = None
- def write(self, b):
- self._write_stack.append(b[:])
- n = self._blocking_script.pop(0)
- if (n < 0):
- raise io.BlockingIOError(0, "test blocking", -n)
- else:
- return n
+ def pop_written(self):
+ s = b"".join(self._write_stack)
+ self._write_stack[:] = []
+ return s
+
+ def block_on(self, char):
+ """Block when a given char is encountered."""
+ self._blocker_char = char
+
+ def readable(self):
+ return True
+
+ def seekable(self):
+ return True
def writable(self):
return True
+ def write(self, b):
+ b = bytes(b)
+ n = -1
+ if self._blocker_char:
+ try:
+ n = b.index(self._blocker_char)
+ except ValueError:
+ pass
+ else:
+ self._blocker_char = None
+ self._write_stack.append(b[:n])
+ raise self.BlockingIOError(0, "test blocking", n)
+ self._write_stack.append(b)
+ return len(b)
+
+class CMockNonBlockWriterIO(MockNonBlockWriterIO, io.RawIOBase):
+ BlockingIOError = io.BlockingIOError
+
+class PyMockNonBlockWriterIO(MockNonBlockWriterIO, pyio.RawIOBase):
+ BlockingIOError = pyio.BlockingIOError
+
class IOTest(unittest.TestCase):
@@ -151,13 +290,13 @@
self.assertEqual(f.read(2), b"x")
def test_raw_file_io(self):
- f = io.open(support.TESTFN, "wb", buffering=0)
+ f = self.open(support.TESTFN, "wb", buffering=0)
self.assertEqual(f.readable(), False)
self.assertEqual(f.writable(), True)
self.assertEqual(f.seekable(), True)
self.write_ops(f)
f.close()
- f = io.open(support.TESTFN, "rb", buffering=0)
+ f = self.open(support.TESTFN, "rb", buffering=0)
self.assertEqual(f.readable(), True)
self.assertEqual(f.writable(), False)
self.assertEqual(f.seekable(), True)
@@ -165,13 +304,13 @@
f.close()
def test_buffered_file_io(self):
- f = io.open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
self.assertEqual(f.readable(), False)
self.assertEqual(f.writable(), True)
self.assertEqual(f.seekable(), True)
self.write_ops(f)
f.close()
- f = io.open(support.TESTFN, "rb")
+ f = self.open(support.TESTFN, "rb")
self.assertEqual(f.readable(), True)
self.assertEqual(f.writable(), False)
self.assertEqual(f.seekable(), True)
@@ -180,22 +319,23 @@
def test_readline(self):
f = io.open(support.TESTFN, "wb")
- f.write(b"abc\ndef\nxyzzy\nfoo")
+ f.write(b"abc\ndef\nxyzzy\nfoo\x00bar\nanother line")
f.close()
- f = io.open(support.TESTFN, "rb")
+ f = self.open(support.TESTFN, "rb")
self.assertEqual(f.readline(), b"abc\n")
self.assertEqual(f.readline(10), b"def\n")
self.assertEqual(f.readline(2), b"xy")
self.assertEqual(f.readline(4), b"zzy\n")
- self.assertEqual(f.readline(), b"foo")
+ self.assertEqual(f.readline(), b"foo\x00bar\n")
+ self.assertEqual(f.readline(), b"another line")
f.close()
def test_raw_bytes_io(self):
- f = io.BytesIO()
+ f = self.BytesIO()
self.write_ops(f)
data = f.getvalue()
self.assertEqual(data, b"hello world\n")
- f = io.BytesIO(data)
+ f = self.BytesIO(data)
self.read_ops(f, True)
def test_large_file_ops(self):
@@ -211,12 +351,10 @@
print("Use 'regrtest.py -u largefile test_io' to run it.",
file=sys.stderr)
return
- f = io.open(support.TESTFN, "w+b", 0)
- self.large_file_ops(f)
- f.close()
- f = io.open(support.TESTFN, "w+b")
- self.large_file_ops(f)
- f.close()
+ with self.open(support.TESTFN, "w+b", 0) as f:
+ self.large_file_ops(f)
+ with self.open(support.TESTFN, "w+b") as f:
+ self.large_file_ops(f)
def test_with_open(self):
for bufsize in (0, 1, 100):
@@ -235,59 +373,107 @@
# issue 5008
def test_append_mode_tell(self):
- with io.open(support.TESTFN, "wb") as f:
+ with self.open(support.TESTFN, "wb") as f:
f.write(b"xxx")
- with io.open(support.TESTFN, "ab", buffering=0) as f:
+ with self.open(support.TESTFN, "ab", buffering=0) as f:
self.assertEqual(f.tell(), 3)
- with io.open(support.TESTFN, "ab") as f:
+ with self.open(support.TESTFN, "ab") as f:
self.assertEqual(f.tell(), 3)
- with io.open(support.TESTFN, "a") as f:
+ with self.open(support.TESTFN, "a") as f:
self.assert_(f.tell() > 0)
def test_destructor(self):
record = []
- class MyFileIO(io.FileIO):
+ class MyFileIO(self.FileIO):
def __del__(self):
record.append(1)
- io.FileIO.__del__(self)
+ try:
+ f = super().__del__
+ except AttributeError:
+ pass
+ else:
+ f()
def close(self):
record.append(2)
- io.FileIO.close(self)
+ super().close()
def flush(self):
record.append(3)
- io.FileIO.flush(self)
- f = MyFileIO(support.TESTFN, "w")
- f.write("xxx")
+ super().flush()
+ f = MyFileIO(support.TESTFN, "wb")
+ f.write(b"xxx")
del f
self.assertEqual(record, [1, 2, 3])
+ f = open(support.TESTFN, "rb")
+ self.assertEqual(f.read(), b"xxx")
+
+ def _check_base_destructor(self, base):
+ record = []
+ class MyIO(base):
+ def __init__(self):
+ # This exercises the availability of attributes on object
+ # destruction.
+ # (in the C version, close() is called by the tp_dealloc
+ # function, not by __del__)
+ self.on_del = 1
+ self.on_close = 2
+ self.on_flush = 3
+ def __del__(self):
+ record.append(self.on_del)
+ try:
+ f = super().__del__
+ except AttributeError:
+ pass
+ else:
+ f()
+ def close(self):
+ record.append(self.on_close)
+ super().close()
+ def flush(self):
+ record.append(self.on_flush)
+ super().flush()
+ f = MyIO()
+ del f
+ self.assertEqual(record, [1, 2, 3])
+
+ def test_IOBase_destructor(self):
+ self._check_base_destructor(self.IOBase)
+
+ def test_RawIOBase_destructor(self):
+ self._check_base_destructor(self.RawIOBase)
+
+ def test_BufferedIOBase_destructor(self):
+ self._check_base_destructor(self.BufferedIOBase)
+
+ def test_TextIOBase_destructor(self):
+ self._check_base_destructor(self.TextIOBase)
def test_close_flushes(self):
- f = io.open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
f.write(b"xxx")
f.close()
- f = io.open(support.TESTFN, "rb")
+ f = self.open(support.TESTFN, "rb")
self.assertEqual(f.read(), b"xxx")
f.close()
def test_array_writes(self):
a = array.array('i', range(10))
n = len(a.tostring())
- f = io.open(support.TESTFN, "wb", 0)
+ f = self.open(support.TESTFN, "wb", 0)
self.assertEqual(f.write(a), n)
f.close()
- f = io.open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
self.assertEqual(f.write(a), n)
f.close()
def test_closefd(self):
- self.assertRaises(ValueError, io.open, support.TESTFN, 'w',
+ self.assertRaises(ValueError, self.open, support.TESTFN, 'w',
closefd=False)
- def testReadClosed(self):
- with io.open(support.TESTFN, "w") as f:
+ def test_read_closed(self):
+ with self.open(support.TESTFN, "w") as f:
f.write("egg\n")
- with io.open(support.TESTFN, "r") as f:
- file = io.open(f.fileno(), "r", closefd=False)
+ with self.open(support.TESTFN, "r") as f:
+ file = self.open(f.fileno(), "r", closefd=False)
self.assertEqual(file.read(), "egg\n")
file.seek(0)
file.close()
@@ -295,85 +481,168 @@
def test_no_closefd_with_filename(self):
# can't use closefd in combination with a file name
- self.assertRaises(ValueError, io.open, support.TESTFN, "r", closefd=False)
+ self.assertRaises(ValueError, self.open, support.TESTFN, "r", closefd=False)
def test_closefd_attr(self):
- with io.open(support.TESTFN, "wb") as f:
+ with self.open(support.TESTFN, "wb") as f:
f.write(b"egg\n")
- with io.open(support.TESTFN, "r") as f:
+ with self.open(support.TESTFN, "r") as f:
self.assertEqual(f.buffer.raw.closefd, True)
- file = io.open(f.fileno(), "r", closefd=False)
+ file = self.open(f.fileno(), "r", closefd=False)
self.assertEqual(file.buffer.raw.closefd, False)
+ def test_garbage_collection(self):
+ # FileIO objects are collected, and collecting them flushes
+ # all data to disk.
+ f = self.FileIO(support.TESTFN, "wb")
+ f.write(b"abcxxx")
+ f.f = f
+ wr = weakref.ref(f)
+ del f
+ gc.collect()
+ self.assert_(wr() is None, wr)
+ with open(support.TESTFN, "rb") as f:
+ self.assertEqual(f.read(), b"abcxxx")
-class MemorySeekTestMixin:
-
- def testInit(self):
- buf = self.buftype("1234567890")
- bytesIo = self.ioclass(buf)
-
- def testRead(self):
- buf = self.buftype("1234567890")
- bytesIo = self.ioclass(buf)
-
- self.assertEquals(buf[:1], bytesIo.read(1))
- self.assertEquals(buf[1:5], bytesIo.read(4))
- self.assertEquals(buf[5:], bytesIo.read(900))
- self.assertEquals(self.EOF, bytesIo.read())
-
- def testReadNoArgs(self):
- buf = self.buftype("1234567890")
- bytesIo = self.ioclass(buf)
-
- self.assertEquals(buf, bytesIo.read())
- self.assertEquals(self.EOF, bytesIo.read())
-
- def testSeek(self):
- buf = self.buftype("1234567890")
- bytesIo = self.ioclass(buf)
-
- bytesIo.read(5)
- bytesIo.seek(0)
- self.assertEquals(buf, bytesIo.read())
-
- bytesIo.seek(3)
- self.assertEquals(buf[3:], bytesIo.read())
- self.assertRaises(TypeError, bytesIo.seek, 0.0)
-
- def testTell(self):
- buf = self.buftype("1234567890")
- bytesIo = self.ioclass(buf)
+class CIOTest(IOTest):
+ pass
- self.assertEquals(0, bytesIo.tell())
- bytesIo.seek(5)
- self.assertEquals(5, bytesIo.tell())
- bytesIo.seek(10000)
- self.assertEquals(10000, bytesIo.tell())
+class PyIOTest(IOTest):
+ pass
-class BytesIOTest(MemorySeekTestMixin, unittest.TestCase):
- @staticmethod
- def buftype(s):
- return s.encode("utf-8")
- ioclass = io.BytesIO
- EOF = b""
+class CommonBufferedTests:
+ # Tests common to BufferedReader, BufferedWriter and BufferedRandom
+ def test_fileno(self):
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
-class StringIOTest(MemorySeekTestMixin, unittest.TestCase):
- buftype = str
- ioclass = io.StringIO
- EOF = ""
+ self.assertEquals(42, bufio.fileno())
+ def test_no_fileno(self):
+ # XXX will we always have fileno() function? If so, kill
+ # this test. Else, write it.
+ pass
-class BufferedReaderTest(unittest.TestCase):
+ def test_invalid_args(self):
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ # Invalid whence
+ self.assertRaises(ValueError, bufio.seek, 0, -1)
+ self.assertRaises(ValueError, bufio.seek, 0, 3)
- def testRead(self):
- rawio = MockRawIO((b"abc", b"d", b"efg"))
- bufio = io.BufferedReader(rawio)
+ def test_override_destructor(self):
+ tp = self.tp
+ record = []
+ class MyBufferedIO(tp):
+ def __del__(self):
+ record.append(1)
+ try:
+ f = super().__del__
+ except AttributeError:
+ pass
+ else:
+ f()
+ def close(self):
+ record.append(2)
+ super().close()
+ def flush(self):
+ record.append(3)
+ super().flush()
+ rawio = self.MockRawIO()
+ bufio = MyBufferedIO(rawio)
+ writable = bufio.writable()
+ del bufio
+ if writable:
+ self.assertEqual(record, [1, 2, 3])
+ else:
+ self.assertEqual(record, [1, 2])
+ def test_context_manager(self):
+ # Test usability as a context manager
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ def _with():
+ with bufio:
+ pass
+ _with()
+ # bufio should now be closed, and using it a second time should raise
+ # a ValueError.
+ self.assertRaises(ValueError, _with)
+
+ def test_error_through_destructor(self):
+ # Test that the exception state is not modified by a destructor,
+ # even if close() fails.
+ rawio = self.CloseFailureIO()
+ def f():
+ self.tp(rawio).xyzzy
+ with support.captured_output("stderr") as s:
+ self.assertRaises(AttributeError, f)
+ s = s.getvalue().strip()
+ if s:
+ # The destructor *may* have printed an unraisable error, check it
+ self.assertEqual(len(s.splitlines()), 1)
+ self.assert_(s.startswith("Exception IOError: "), s)
+ self.assert_(s.endswith(" ignored"), s)
+
+
+class BufferedReaderTest(unittest.TestCase, CommonBufferedTests):
+ read_mode = "rb"
+
+ def test_constructor(self):
+ rawio = self.MockRawIO([b"abc"])
+ bufio = self.tp(rawio)
+ bufio.__init__(rawio)
+ bufio.__init__(rawio, buffer_size=1024)
+ bufio.__init__(rawio, buffer_size=16)
+ self.assertEquals(b"abc", bufio.read())
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=0)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-16)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-1)
+ rawio = self.MockRawIO([b"abc"])
+ bufio.__init__(rawio)
+ self.assertEquals(b"abc", bufio.read())
+
+ def test_read(self):
+ rawio = self.MockRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
self.assertEquals(b"abcdef", bufio.read(6))
+ # Invalid args
+ self.assertRaises(ValueError, bufio.read, -2)
- def testBuffering(self):
+ def test_read1(self):
+ rawio = self.MockRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
+ self.assertEquals(b"a", bufio.read(1))
+ self.assertEquals(b"b", bufio.read1(1))
+ self.assertEquals(rawio._reads, 1)
+ self.assertEquals(b"c", bufio.read1(100))
+ self.assertEquals(rawio._reads, 1)
+ self.assertEquals(b"d", bufio.read1(100))
+ self.assertEquals(rawio._reads, 2)
+ self.assertEquals(b"efg", bufio.read1(100))
+ self.assertEquals(rawio._reads, 3)
+ self.assertEquals(b"", bufio.read1(100))
+ # Invalid args
+ self.assertRaises(ValueError, bufio.read1, -1)
+
+ def test_readinto(self):
+ rawio = self.MockRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
+ b = bytearray(2)
+ self.assertEquals(bufio.readinto(b), 2)
+ self.assertEquals(b, b"ab")
+ self.assertEquals(bufio.readinto(b), 2)
+ self.assertEquals(b, b"cd")
+ self.assertEquals(bufio.readinto(b), 2)
+ self.assertEquals(b, b"ef")
+ self.assertEquals(bufio.readinto(b), 1)
+ self.assertEquals(b, b"gf")
+ self.assertEquals(bufio.readinto(b), 0)
+ self.assertEquals(b, b"gf")
+
+ def test_buffering(self):
data = b"abcdefghi"
dlen = len(data)
@@ -384,49 +653,40 @@
]
for bufsize, buf_read_sizes, raw_read_sizes in tests:
- rawio = MockFileIO(data)
- bufio = io.BufferedReader(rawio, buffer_size=bufsize)
+ rawio = self.MockFileIO(data)
+ bufio = self.tp(rawio, buffer_size=bufsize)
pos = 0
for nbytes in buf_read_sizes:
self.assertEquals(bufio.read(nbytes), data[pos:pos+nbytes])
pos += nbytes
+ # this is mildly implementation-dependent
self.assertEquals(rawio.read_history, raw_read_sizes)
- def testReadNonBlocking(self):
+ def test_read_non_blocking(self):
# Inject some None's in there to simulate EWOULDBLOCK
- rawio = MockRawIO((b"abc", b"d", None, b"efg", None, None))
- bufio = io.BufferedReader(rawio)
+ rawio = self.MockRawIO((b"abc", b"d", None, b"efg", None, None, None))
+ bufio = self.tp(rawio)
self.assertEquals(b"abcd", bufio.read(6))
self.assertEquals(b"e", bufio.read(1))
self.assertEquals(b"fg", bufio.read())
+ self.assertEquals(b"", bufio.peek(1))
self.assert_(None is bufio.read())
self.assertEquals(b"", bufio.read())
- def testReadToEof(self):
- rawio = MockRawIO((b"abc", b"d", b"efg"))
- bufio = io.BufferedReader(rawio)
+ def test_read_past_eof(self):
+ rawio = self.MockRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
self.assertEquals(b"abcdefg", bufio.read(9000))
- def testReadNoArgs(self):
- rawio = MockRawIO((b"abc", b"d", b"efg"))
- bufio = io.BufferedReader(rawio)
+ def test_read_all(self):
+ rawio = self.MockRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
self.assertEquals(b"abcdefg", bufio.read())
- def testFileno(self):
- rawio = MockRawIO((b"abc", b"d", b"efg"))
- bufio = io.BufferedReader(rawio)
-
- self.assertEquals(42, bufio.fileno())
-
- def testFilenoNoFileno(self):
- # XXX will we always have fileno() function? If so, kill
- # this test. Else, write it.
- pass
-
- def testThreads(self):
+ def test_threads(self):
try:
# Write out many bytes with exactly the same number of 0's,
# 1's... 255's. This will help us check that concurrent reading
@@ -437,8 +697,8 @@
s = bytes(bytearray(l))
with io.open(support.TESTFN, "wb") as f:
f.write(s)
- with io.open(support.TESTFN, "rb", buffering=0) as raw:
- bufio = io.BufferedReader(raw, 8)
+ with io.open(support.TESTFN, self.read_mode, buffering=0) as raw:
+ bufio = self.tp(raw, 8)
errors = []
results = []
def f():
@@ -468,80 +728,230 @@
finally:
support.unlink(support.TESTFN)
+ def test_misbehaved_io(self):
+ rawio = self.MisbehavedRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
+ self.assertRaises(IOError, bufio.seek, 0)
+ self.assertRaises(IOError, bufio.tell)
+
+class CBufferedReaderTest(BufferedReaderTest):
+ tp = io.BufferedReader
+
+ def test_constructor(self):
+ BufferedReaderTest.test_constructor(self)
+ # The allocation can succeed on 32-bit builds, e.g. with more
+ # than 2GB RAM and a 64-bit kernel.
+ if sys.maxsize > 0x7FFFFFFF:
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ self.assertRaises((OverflowError, MemoryError, ValueError),
+ bufio.__init__, rawio, sys.maxsize)
+
+ def test_initialization(self):
+ rawio = self.MockRawIO([b"abc"])
+ bufio = self.tp(rawio)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=0)
+ self.assertRaises(ValueError, bufio.read)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-16)
+ self.assertRaises(ValueError, bufio.read)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-1)
+ self.assertRaises(ValueError, bufio.read)
+
+ def test_misbehaved_io_read(self):
+ rawio = self.MisbehavedRawIO((b"abc", b"d", b"efg"))
+ bufio = self.tp(rawio)
+ # _pyio.BufferedReader seems to implement reading different, so that
+ # checking this is not so easy.
+ self.assertRaises(IOError, bufio.read, 10)
+
+ def test_garbage_collection(self):
+ # C BufferedReader objects are collected.
+ # The Python version has __del__, so it ends into gc.garbage instead
+ rawio = self.FileIO(support.TESTFN, "w+b")
+ f = self.tp(rawio)
+ f.f = f
+ wr = weakref.ref(f)
+ del f
+ gc.collect()
+ self.assert_(wr() is None, wr)
+class PyBufferedReaderTest(BufferedReaderTest):
+ tp = pyio.BufferedReader
-class BufferedWriterTest(unittest.TestCase):
- def testWrite(self):
- # Write to the buffered IO but don't overflow the buffer.
- writer = MockRawIO()
- bufio = io.BufferedWriter(writer, 8)
+class BufferedWriterTest(unittest.TestCase, CommonBufferedTests):
+ write_mode = "wb"
+
+ def test_constructor(self):
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ bufio.__init__(rawio)
+ bufio.__init__(rawio, buffer_size=1024)
+ bufio.__init__(rawio, buffer_size=16)
+ self.assertEquals(3, bufio.write(b"abc"))
+ bufio.flush()
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=0)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-16)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-1)
+ bufio.__init__(rawio)
+ self.assertEquals(3, bufio.write(b"ghi"))
+ bufio.flush()
+ self.assertEquals(b"".join(rawio._write_stack), b"abcghi")
+ def test_write(self):
+ # Write to the buffered IO but don't overflow the buffer.
+ writer = self.MockRawIO()
+ bufio = self.tp(writer, 8)
bufio.write(b"abc")
-
self.assertFalse(writer._write_stack)
- def testWriteOverflow(self):
- writer = MockRawIO()
- bufio = io.BufferedWriter(writer, 8)
-
- bufio.write(b"abc")
- bufio.write(b"defghijkl")
-
- self.assertEquals(b"abcdefghijkl", writer._write_stack[0])
-
- def testWriteNonBlocking(self):
- raw = MockNonBlockWriterIO((9, 2, 22, -6, 10, 12, 12))
- bufio = io.BufferedWriter(raw, 8, 16)
-
- bufio.write(b"asdf")
- bufio.write(b"asdfa")
- self.assertEquals(b"asdfasdfa", raw._write_stack[0])
-
- bufio.write(b"asdfasdfasdf")
- self.assertEquals(b"asdfasdfasdf", raw._write_stack[1])
- bufio.write(b"asdfasdfasdf")
- self.assertEquals(b"dfasdfasdf", raw._write_stack[2])
- self.assertEquals(b"asdfasdfasdf", raw._write_stack[3])
-
- bufio.write(b"asdfasdfasdf")
+ def test_write_overflow(self):
+ writer = self.MockRawIO()
+ bufio = self.tp(writer, 8)
+ contents = b"abcdefghijklmnop"
+ for n in range(0, len(contents), 3):
+ bufio.write(contents[n:n+3])
+ flushed = b"".join(writer._write_stack)
+ # At least (total - 8) bytes were implicitly flushed, perhaps more
+ # depending on the implementation.
+ self.assert_(flushed.startswith(contents[:-8]), flushed)
+
+ def check_writes(self, intermediate_func):
+ # Lots of writes, test the flushed output is as expected.
+ contents = bytes(range(256)) * 1000
+ n = 0
+ writer = self.MockRawIO()
+ bufio = self.tp(writer, 13)
+ # Generator of write sizes: repeat each N 15 times then proceed to N+1
+ def gen_sizes():
+ for size in count(1):
+ for i in range(15):
+ yield size
+ sizes = gen_sizes()
+ while n < len(contents):
+ size = min(next(sizes), len(contents) - n)
+ self.assertEquals(bufio.write(contents[n:n+size]), size)
+ intermediate_func(bufio)
+ n += size
+ bufio.flush()
+ self.assertEquals(contents, b"".join(writer._write_stack))
- # XXX I don't like this test. It relies too heavily on how the
- # algorithm actually works, which we might change. Refactor
- # later.
+ def test_writes(self):
+ self.check_writes(lambda bufio: None)
- def testFileno(self):
- rawio = MockRawIO((b"abc", b"d", b"efg"))
- bufio = io.BufferedWriter(rawio)
+ def test_writes_and_flushes(self):
+ self.check_writes(lambda bufio: bufio.flush())
- self.assertEquals(42, bufio.fileno())
+ def test_writes_and_seeks(self):
+ def _seekabs(bufio):
+ pos = bufio.tell()
+ bufio.seek(pos + 1, 0)
+ bufio.seek(pos - 1, 0)
+ bufio.seek(pos, 0)
+ self.check_writes(_seekabs)
+ def _seekrel(bufio):
+ pos = bufio.seek(0, 1)
+ bufio.seek(+1, 1)
+ bufio.seek(-1, 1)
+ bufio.seek(pos, 0)
+ self.check_writes(_seekrel)
+
+ def test_writes_and_truncates(self):
+ self.check_writes(lambda bufio: bufio.truncate(bufio.tell()))
+
+ def test_write_non_blocking(self):
+ raw = self.MockNonBlockWriterIO()
+ bufio = self.tp(raw, 8, 8)
+
+ self.assertEquals(bufio.write(b"abcd"), 4)
+ self.assertEquals(bufio.write(b"efghi"), 5)
+ # 1 byte will be written, the rest will be buffered
+ raw.block_on(b"k")
+ self.assertEquals(bufio.write(b"jklmn"), 5)
- def testFlush(self):
- writer = MockRawIO()
- bufio = io.BufferedWriter(writer, 8)
+ # 8 bytes will be written, 8 will be buffered and the rest will be lost
+ raw.block_on(b"0")
+ try:
+ bufio.write(b"opqrwxyz0123456789")
+ except self.BlockingIOError as e:
+ written = e.characters_written
+ else:
+ self.fail("BlockingIOError should have been raised")
+ self.assertEquals(written, 16)
+ self.assertEquals(raw.pop_written(),
+ b"abcdefghijklmnopqrwxyz")
+
+ self.assertEquals(bufio.write(b"ABCDEFGHI"), 9)
+ s = raw.pop_written()
+ # Previously buffered bytes were flushed
+ self.assertTrue(s.startswith(b"01234567A"), s)
+
+ def test_write_and_rewind(self):
+ raw = io.BytesIO()
+ bufio = self.tp(raw, 4)
+ self.assertEqual(bufio.write(b"abcdef"), 6)
+ self.assertEqual(bufio.tell(), 6)
+ bufio.seek(0, 0)
+ self.assertEqual(bufio.write(b"XY"), 2)
+ bufio.seek(6, 0)
+ self.assertEqual(raw.getvalue(), b"XYcdef")
+ self.assertEqual(bufio.write(b"123456"), 6)
+ bufio.flush()
+ self.assertEqual(raw.getvalue(), b"XYcdef123456")
+ def test_flush(self):
+ writer = self.MockRawIO()
+ bufio = self.tp(writer, 8)
bufio.write(b"abc")
bufio.flush()
+ self.assertEquals(b"abc", writer._write_stack[0])
+ def test_destructor(self):
+ writer = self.MockRawIO()
+ bufio = self.tp(writer, 8)
+ bufio.write(b"abc")
+ del bufio
self.assertEquals(b"abc", writer._write_stack[0])
- def testThreads(self):
- # BufferedWriter should not raise exceptions or crash
- # when called from multiple threads.
+ def test_truncate(self):
+ # Truncate implicitly flushes the buffer.
+ with io.open(support.TESTFN, self.write_mode, buffering=0) as raw:
+ bufio = self.tp(raw, 8)
+ bufio.write(b"abcdef")
+ self.assertEqual(bufio.truncate(3), 3)
+ self.assertEqual(bufio.tell(), 3)
+ with io.open(support.TESTFN, "rb", buffering=0) as f:
+ self.assertEqual(f.read(), b"abc")
+
+ def test_threads(self):
try:
+ # Write out many bytes from many threads and test they were
+ # all flushed.
+ N = 1000
+ contents = bytes(range(256)) * N
+ sizes = cycle([1, 19])
+ n = 0
+ queue = deque()
+ while n < len(contents):
+ size = next(sizes)
+ queue.append(contents[n:n+size])
+ n += size
+ del contents
# We use a real file object because it allows us to
# exercise situations where the GIL is released before
# writing the buffer to the raw streams. This is in addition
# to concurrency issues due to switching threads in the middle
# of Python code.
- with io.open(support.TESTFN, "wb", buffering=0) as raw:
- bufio = io.BufferedWriter(raw, 8)
+ with io.open(support.TESTFN, self.write_mode, buffering=0) as raw:
+ bufio = self.tp(raw, 8)
errors = []
def f():
try:
- # Write enough bytes to flush the buffer
- s = b"a" * 19
- for i in range(50):
+ while True:
+ try:
+ s = queue.popleft()
+ except IndexError:
+ return
bufio.write(s)
except Exception as e:
errors.append(e)
@@ -554,37 +964,102 @@
t.join()
self.assertFalse(errors,
"the following exceptions were caught: %r" % errors)
+ bufio.close()
+ with io.open(support.TESTFN, "rb") as f:
+ s = f.read()
+ for i in range(256):
+ self.assertEquals(s.count(bytes([i])), N)
finally:
support.unlink(support.TESTFN)
+ def test_misbehaved_io(self):
+ rawio = self.MisbehavedRawIO()
+ bufio = self.tp(rawio, 5)
+ self.assertRaises(IOError, bufio.seek, 0)
+ self.assertRaises(IOError, bufio.tell)
+ self.assertRaises(IOError, bufio.write, b"abcdef")
+
+class CBufferedWriterTest(BufferedWriterTest):
+ tp = io.BufferedWriter
+
+ def test_constructor(self):
+ BufferedWriterTest.test_constructor(self)
+ # The allocation can succeed on 32-bit builds, e.g. with more
+ # than 2GB RAM and a 64-bit kernel.
+ if sys.maxsize > 0x7FFFFFFF:
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ self.assertRaises((OverflowError, MemoryError, ValueError),
+ bufio.__init__, rawio, sys.maxsize)
+
+ def test_initialization(self):
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=0)
+ self.assertRaises(ValueError, bufio.write, b"def")
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-16)
+ self.assertRaises(ValueError, bufio.write, b"def")
+ self.assertRaises(ValueError, bufio.__init__, rawio, buffer_size=-1)
+ self.assertRaises(ValueError, bufio.write, b"def")
+
+ def test_garbage_collection(self):
+ # C BufferedWriter objects are collected, and collecting them flushes
+ # all data to disk.
+ # The Python version has __del__, so it ends into gc.garbage instead
+ rawio = self.FileIO(support.TESTFN, "w+b")
+ f = self.tp(rawio)
+ f.write(b"123xxx")
+ f.x = f
+ wr = weakref.ref(f)
+ del f
+ gc.collect()
+ self.assert_(wr() is None, wr)
+ with open(support.TESTFN, "rb") as f:
+ self.assertEqual(f.read(), b"123xxx")
+
+
+class PyBufferedWriterTest(BufferedWriterTest):
+ tp = pyio.BufferedWriter
class BufferedRWPairTest(unittest.TestCase):
- def testRWPair(self):
- r = MockRawIO(())
- w = MockRawIO()
- pair = io.BufferedRWPair(r, w)
+ def test_basic(self):
+ r = self.MockRawIO(())
+ w = self.MockRawIO()
+ pair = self.tp(r, w)
self.assertFalse(pair.closed)
# XXX More Tests
+class CBufferedRWPairTest(BufferedRWPairTest):
+ tp = io.BufferedRWPair
+
+class PyBufferedRWPairTest(BufferedRWPairTest):
+ tp = pyio.BufferedRWPair
+
+
+class BufferedRandomTest(BufferedReaderTest, BufferedWriterTest):
+ read_mode = "rb+"
+ write_mode = "wb+"
-class BufferedRandomTest(unittest.TestCase):
+ def test_constructor(self):
+ BufferedReaderTest.test_constructor(self)
+ BufferedWriterTest.test_constructor(self)
- def testReadAndWrite(self):
- raw = MockRawIO((b"asdf", b"ghjk"))
- rw = io.BufferedRandom(raw, 8, 12)
+ def test_read_and_write(self):
+ raw = self.MockRawIO((b"asdf", b"ghjk"))
+ rw = self.tp(raw, 8, 12)
self.assertEqual(b"as", rw.read(2))
rw.write(b"ddd")
rw.write(b"eee")
self.assertFalse(raw._write_stack) # Buffer writes
- self.assertEqual(b"ghjk", rw.read()) # This read forces write flush
+ self.assertEqual(b"ghjk", rw.read())
self.assertEquals(b"dddeee", raw._write_stack[0])
- def testSeekAndTell(self):
- raw = io.BytesIO(b"asdfghjkl")
- rw = io.BufferedRandom(raw)
+ def test_seek_and_tell(self):
+ raw = self.BytesIO(b"asdfghjkl")
+ rw = self.tp(raw)
self.assertEquals(b"as", rw.read(2))
self.assertEquals(2, rw.tell())
@@ -602,6 +1077,115 @@
self.assertEquals(b"fl", rw.read(11))
self.assertRaises(TypeError, rw.seek, 0.0)
+ def check_flush_and_read(self, read_func):
+ raw = self.BytesIO(b"abcdefghi")
+ bufio = self.tp(raw)
+
+ self.assertEquals(b"ab", read_func(bufio, 2))
+ bufio.write(b"12")
+ self.assertEquals(b"ef", read_func(bufio, 2))
+ self.assertEquals(6, bufio.tell())
+ bufio.flush()
+ self.assertEquals(6, bufio.tell())
+ self.assertEquals(b"ghi", read_func(bufio))
+ raw.seek(0, 0)
+ raw.write(b"XYZ")
+ # flush() resets the read buffer
+ bufio.flush()
+ bufio.seek(0, 0)
+ self.assertEquals(b"XYZ", read_func(bufio, 3))
+
+ def test_flush_and_read(self):
+ self.check_flush_and_read(lambda bufio, *args: bufio.read(*args))
+
+ def test_flush_and_readinto(self):
+ def _readinto(bufio, n=-1):
+ b = bytearray(n if n >= 0 else 9999)
+ n = bufio.readinto(b)
+ return bytes(b[:n])
+ self.check_flush_and_read(_readinto)
+
+ def test_flush_and_peek(self):
+ def _peek(bufio, n=-1):
+ # This relies on the fact that the buffer can contain the whole
+ # raw stream, otherwise peek() can return less.
+ b = bufio.peek(n)
+ if n != -1:
+ b = b[:n]
+ bufio.seek(len(b), 1)
+ return b
+ self.check_flush_and_read(_peek)
+
+ def test_flush_and_write(self):
+ raw = self.BytesIO(b"abcdefghi")
+ bufio = self.tp(raw)
+
+ bufio.write(b"123")
+ bufio.flush()
+ bufio.write(b"45")
+ bufio.flush()
+ bufio.seek(0, 0)
+ self.assertEquals(b"12345fghi", raw.getvalue())
+ self.assertEquals(b"12345fghi", bufio.read())
+
+ def test_threads(self):
+ BufferedReaderTest.test_threads(self)
+ BufferedWriterTest.test_threads(self)
+
+ def test_writes_and_peek(self):
+ def _peek(bufio):
+ bufio.peek(1)
+ self.check_writes(_peek)
+ def _peek(bufio):
+ pos = bufio.tell()
+ bufio.seek(-1, 1)
+ bufio.peek(1)
+ bufio.seek(pos, 0)
+ self.check_writes(_peek)
+
+ def test_writes_and_reads(self):
+ def _read(bufio):
+ bufio.seek(-1, 1)
+ bufio.read(1)
+ self.check_writes(_read)
+
+ def test_writes_and_read1s(self):
+ def _read1(bufio):
+ bufio.seek(-1, 1)
+ bufio.read1(1)
+ self.check_writes(_read1)
+
+ def test_writes_and_readintos(self):
+ def _read(bufio):
+ bufio.seek(-1, 1)
+ bufio.readinto(bytearray(1))
+ self.check_writes(_read)
+
+ def test_misbehaved_io(self):
+ BufferedReaderTest.test_misbehaved_io(self)
+ BufferedWriterTest.test_misbehaved_io(self)
+
+class CBufferedRandomTest(BufferedRandomTest):
+ tp = io.BufferedRandom
+
+ def test_constructor(self):
+ BufferedRandomTest.test_constructor(self)
+ # The allocation can succeed on 32-bit builds, e.g. with more
+ # than 2GB RAM and a 64-bit kernel.
+ if sys.maxsize > 0x7FFFFFFF:
+ rawio = self.MockRawIO()
+ bufio = self.tp(rawio)
+ self.assertRaises((OverflowError, MemoryError, ValueError),
+ bufio.__init__, rawio, sys.maxsize)
+
+ def test_garbage_collection(self):
+ CBufferedReaderTest.test_garbage_collection(self)
+ CBufferedWriterTest.test_garbage_collection(self)
+
+class PyBufferedRandomTest(BufferedRandomTest):
+ tp = pyio.BufferedRandom
+
+
# To fully exercise seek/tell, the StatefulIncrementalDecoder has these
# properties:
# - A single output character can correspond to many bytes of input.
@@ -735,7 +1319,7 @@
'm--------------.')
]
- def testDecoder(self):
+ def test_decoder(self):
# Try a few one-shot test cases.
for input, eof, output in self.test_cases:
d = StatefulIncrementalDecoder()
@@ -756,10 +1340,24 @@
def tearDown(self):
support.unlink(support.TESTFN)
- def testLineBuffering(self):
- r = io.BytesIO()
- b = io.BufferedWriter(r, 1000)
- t = io.TextIOWrapper(b, newline="\n", line_buffering=True)
+ def test_constructor(self):
+ r = self.BytesIO(b"\xc3\xa9\n\n")
+ b = self.BufferedReader(r, 1000)
+ t = self.TextIOWrapper(b)
+ t.__init__(b, encoding="latin1", newline="\r\n")
+ self.assertEquals(t.encoding, "latin1")
+ self.assertEquals(t.line_buffering, False)
+ t.__init__(b, encoding="utf8", line_buffering=True)
+ self.assertEquals(t.encoding, "utf8")
+ self.assertEquals(t.line_buffering, True)
+ self.assertEquals("\xe9\n", t.readline())
+ self.assertRaises(TypeError, t.__init__, b, newline=42)
+ self.assertRaises(ValueError, t.__init__, b, newline='xyzzy')
+
+ def test_line_buffering(self):
+ r = self.BytesIO()
+ b = self.BufferedWriter(r, 1000)
+ t = self.TextIOWrapper(b, newline="\n", line_buffering=True)
t.write("X")
self.assertEquals(r.getvalue(), b"") # No flush happened
t.write("Y\nZ")
@@ -767,83 +1365,58 @@
t.write("A\rB")
self.assertEquals(r.getvalue(), b"XY\nZA\rB")
- def testEncodingErrorsReading(self):
+ def test_encoding(self):
+ # Check the encoding attribute is always set, and valid
+ b = self.BytesIO()
+ t = self.TextIOWrapper(b, encoding="utf8")
+ self.assertEqual(t.encoding, "utf8")
+ t = self.TextIOWrapper(b)
+ self.assert_(t.encoding is not None)
+ codecs.lookup(t.encoding)
+
+ def test_encoding_errors_reading(self):
# (1) default
- b = io.BytesIO(b"abc\n\xff\n")
- t = io.TextIOWrapper(b, encoding="ascii")
+ b = self.BytesIO(b"abc\n\xff\n")
+ t = self.TextIOWrapper(b, encoding="ascii")
self.assertRaises(UnicodeError, t.read)
# (2) explicit strict
- b = io.BytesIO(b"abc\n\xff\n")
- t = io.TextIOWrapper(b, encoding="ascii", errors="strict")
+ b = self.BytesIO(b"abc\n\xff\n")
+ t = self.TextIOWrapper(b, encoding="ascii", errors="strict")
self.assertRaises(UnicodeError, t.read)
# (3) ignore
- b = io.BytesIO(b"abc\n\xff\n")
- t = io.TextIOWrapper(b, encoding="ascii", errors="ignore")
+ b = self.BytesIO(b"abc\n\xff\n")
+ t = self.TextIOWrapper(b, encoding="ascii", errors="ignore")
self.assertEquals(t.read(), "abc\n\n")
# (4) replace
- b = io.BytesIO(b"abc\n\xff\n")
- t = io.TextIOWrapper(b, encoding="ascii", errors="replace")
+ b = self.BytesIO(b"abc\n\xff\n")
+ t = self.TextIOWrapper(b, encoding="ascii", errors="replace")
self.assertEquals(t.read(), "abc\n\ufffd\n")
- def testEncodingErrorsWriting(self):
+ def test_encoding_errors_writing(self):
# (1) default
- b = io.BytesIO()
- t = io.TextIOWrapper(b, encoding="ascii")
+ b = self.BytesIO()
+ t = self.TextIOWrapper(b, encoding="ascii")
self.assertRaises(UnicodeError, t.write, "\xff")
# (2) explicit strict
- b = io.BytesIO()
- t = io.TextIOWrapper(b, encoding="ascii", errors="strict")
+ b = self.BytesIO()
+ t = self.TextIOWrapper(b, encoding="ascii", errors="strict")
self.assertRaises(UnicodeError, t.write, "\xff")
# (3) ignore
- b = io.BytesIO()
- t = io.TextIOWrapper(b, encoding="ascii", errors="ignore",
+ b = self.BytesIO()
+ t = self.TextIOWrapper(b, encoding="ascii", errors="ignore",
newline="\n")
t.write("abc\xffdef\n")
t.flush()
self.assertEquals(b.getvalue(), b"abcdef\n")
# (4) replace
- b = io.BytesIO()
- t = io.TextIOWrapper(b, encoding="ascii", errors="replace",
+ b = self.BytesIO()
+ t = self.TextIOWrapper(b, encoding="ascii", errors="replace",
newline="\n")
t.write("abc\xffdef\n")
t.flush()
self.assertEquals(b.getvalue(), b"abc?def\n")
- def testNewlinesInput(self):
- testdata = b"AAA\nBBB\nCCC\rDDD\rEEE\r\nFFF\r\nGGG"
- normalized = testdata.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
- for newline, expected in [
- (None, normalized.decode("ascii").splitlines(True)),
- ("", testdata.decode("ascii").splitlines(True)),
- ("\n", ["AAA\n", "BBB\n", "CCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
- ("\r\n", ["AAA\nBBB\nCCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
- ("\r", ["AAA\nBBB\nCCC\r", "DDD\r", "EEE\r", "\nFFF\r", "\nGGG"]),
- ]:
- buf = io.BytesIO(testdata)
- txt = io.TextIOWrapper(buf, encoding="ascii", newline=newline)
- self.assertEquals(txt.readlines(), expected)
- txt.seek(0)
- self.assertEquals(txt.read(), "".join(expected))
-
- def testNewlinesOutput(self):
- testdict = {
- "": b"AAA\nBBB\nCCC\nX\rY\r\nZ",
- "\n": b"AAA\nBBB\nCCC\nX\rY\r\nZ",
- "\r": b"AAA\rBBB\rCCC\rX\rY\r\rZ",
- "\r\n": b"AAA\r\nBBB\r\nCCC\r\nX\rY\r\r\nZ",
- }
- tests = [(None, testdict[os.linesep])] + sorted(testdict.items())
- for newline, expected in tests:
- buf = io.BytesIO()
- txt = io.TextIOWrapper(buf, encoding="ascii", newline=newline)
- txt.write("AAA\nB")
- txt.write("BB\nCCC\n")
- txt.write("X\rY\r\nZ")
- txt.flush()
- self.assertEquals(buf.closed, False)
- self.assertEquals(buf.getvalue(), expected)
-
- def testNewlines(self):
+ def test_newlines(self):
input_lines = [ "unix\n", "windows\r\n", "os9\r", "last\n", "nonl" ]
tests = [
@@ -867,8 +1440,8 @@
for do_reads in (False, True):
for bufsize in range(1, 10):
for newline, exp_lines in tests:
- bufio = io.BufferedReader(io.BytesIO(data), bufsize)
- textio = io.TextIOWrapper(bufio, newline=newline,
+ bufio = self.BufferedReader(self.BytesIO(data), bufsize)
+ textio = self.TextIOWrapper(bufio, newline=newline,
encoding=encoding)
if do_reads:
got_lines = []
@@ -885,60 +1458,100 @@
self.assertEquals(got_line, exp_line)
self.assertEquals(len(got_lines), len(exp_lines))
- def testNewlinesInput(self):
- testdata = b"AAA\nBBB\nCCC\rDDD\rEEE\r\nFFF\r\nGGG"
+ def test_newlines_input(self):
+ testdata = b"AAA\nBB\x00B\nCCC\rDDD\rEEE\r\nFFF\r\nGGG"
normalized = testdata.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
for newline, expected in [
(None, normalized.decode("ascii").splitlines(True)),
("", testdata.decode("ascii").splitlines(True)),
- ("\n", ["AAA\n", "BBB\n", "CCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
- ("\r\n", ["AAA\nBBB\nCCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
- ("\r", ["AAA\nBBB\nCCC\r", "DDD\r", "EEE\r", "\nFFF\r", "\nGGG"]),
+ ("\n", ["AAA\n", "BB\x00B\n", "CCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
+ ("\r\n", ["AAA\nBB\x00B\nCCC\rDDD\rEEE\r\n", "FFF\r\n", "GGG"]),
+ ("\r", ["AAA\nBB\x00B\nCCC\r", "DDD\r", "EEE\r", "\nFFF\r", "\nGGG"]),
]:
- buf = io.BytesIO(testdata)
- txt = io.TextIOWrapper(buf, encoding="ascii", newline=newline)
+ buf = self.BytesIO(testdata)
+ txt = self.TextIOWrapper(buf, encoding="ascii", newline=newline)
self.assertEquals(txt.readlines(), expected)
txt.seek(0)
self.assertEquals(txt.read(), "".join(expected))
- def testNewlinesOutput(self):
- data = "AAA\nBBB\rCCC\n"
- data_lf = b"AAA\nBBB\rCCC\n"
- data_cr = b"AAA\rBBB\rCCC\r"
- data_crlf = b"AAA\r\nBBB\rCCC\r\n"
- save_linesep = os.linesep
- try:
- for os.linesep, newline, expected in [
- ("\n", None, data_lf),
- ("\r\n", None, data_crlf),
- ("\n", "", data_lf),
- ("\r\n", "", data_lf),
- ("\n", "\n", data_lf),
- ("\r\n", "\n", data_lf),
- ("\n", "\r", data_cr),
- ("\r\n", "\r", data_cr),
- ("\n", "\r\n", data_crlf),
- ("\r\n", "\r\n", data_crlf),
- ]:
- buf = io.BytesIO()
- txt = io.TextIOWrapper(buf, encoding="ascii", newline=newline)
- txt.write(data)
- txt.close()
- self.assertEquals(buf.closed, True)
- self.assertRaises(ValueError, buf.getvalue)
- finally:
- os.linesep = save_linesep
+ def test_newlines_output(self):
+ testdict = {
+ "": b"AAA\nBBB\nCCC\nX\rY\r\nZ",
+ "\n": b"AAA\nBBB\nCCC\nX\rY\r\nZ",
+ "\r": b"AAA\rBBB\rCCC\rX\rY\r\rZ",
+ "\r\n": b"AAA\r\nBBB\r\nCCC\r\nX\rY\r\r\nZ",
+ }
+ tests = [(None, testdict[os.linesep])] + sorted(testdict.items())
+ for newline, expected in tests:
+ buf = self.BytesIO()
+ txt = self.TextIOWrapper(buf, encoding="ascii", newline=newline)
+ txt.write("AAA\nB")
+ txt.write("BB\nCCC\n")
+ txt.write("X\rY\r\nZ")
+ txt.flush()
+ self.assertEquals(buf.closed, False)
+ self.assertEquals(buf.getvalue(), expected)
+
+ def test_destructor(self):
+ l = []
+ base = self.BytesIO
+ class MyBytesIO(base):
+ def close(self):
+ l.append(self.getvalue())
+ base.close(self)
+ b = MyBytesIO()
+ t = self.TextIOWrapper(b, encoding="ascii")
+ t.write("abc")
+ del t
+ self.assertEquals([b"abc"], l)
+
+ def test_override_destructor(self):
+ record = []
+ class MyTextIO(self.TextIOWrapper):
+ def __del__(self):
+ record.append(1)
+ try:
+ f = super().__del__
+ except AttributeError:
+ pass
+ else:
+ f()
+ def close(self):
+ record.append(2)
+ super().close()
+ def flush(self):
+ record.append(3)
+ super().flush()
+ b = self.BytesIO()
+ t = MyTextIO(b, encoding="ascii")
+ del t
+ self.assertEqual(record, [1, 2, 3])
+
+ def test_error_through_destructor(self):
+ # Test that the exception state is not modified by a destructor,
+ # even if close() fails.
+ rawio = self.CloseFailureIO()
+ def f():
+ self.TextIOWrapper(rawio).xyzzy
+ with support.captured_output("stderr") as s:
+ self.assertRaises(AttributeError, f)
+ s = s.getvalue().strip()
+ if s:
+ # The destructor *may* have printed an unraisable error, check it
+ self.assertEqual(len(s.splitlines()), 1)
+ self.assert_(s.startswith("Exception IOError: "), s)
+ self.assert_(s.endswith(" ignored"), s)
# Systematic tests of the text I/O API
- def testBasicIO(self):
+ def test_basic_io(self):
for chunksize in (1, 2, 3, 4, 5, 15, 16, 17, 31, 32, 33, 63, 64, 65):
for enc in "ascii", "latin1", "utf8" :# , "utf-16-be", "utf-16-le":
- f = io.open(support.TESTFN, "w+", encoding=enc)
+ f = self.open(support.TESTFN, "w+", encoding=enc)
f._CHUNK_SIZE = chunksize
self.assertEquals(f.write("abc"), 3)
f.close()
- f = io.open(support.TESTFN, "r+", encoding=enc)
+ f = self.open(support.TESTFN, "r+", encoding=enc)
f._CHUNK_SIZE = chunksize
self.assertEquals(f.tell(), 0)
self.assertEquals(f.read(), "abc")
@@ -980,8 +1593,8 @@
rlines.append((pos, line))
self.assertEquals(rlines, wlines)
- def testTelling(self):
- f = io.open(support.TESTFN, "w+", encoding="utf8")
+ def test_telling(self):
+ f = self.open(support.TESTFN, "w+", encoding="utf8")
p0 = f.tell()
f.write("\xff\n")
p1 = f.tell()
@@ -1000,8 +1613,8 @@
self.assertEquals(f.tell(), p2)
f.close()
- def testSeeking(self):
- chunk_size = io.TextIOWrapper._CHUNK_SIZE
+ def test_seeking(self):
+ chunk_size = _default_chunk_size()
prefix_size = chunk_size - 2
u_prefix = "a" * prefix_size
prefix = bytes(u_prefix.encode("utf-8"))
@@ -1009,48 +1622,46 @@
u_suffix = "\u8888\n"
suffix = bytes(u_suffix.encode("utf-8"))
line = prefix + suffix
- f = io.open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
f.write(line*2)
f.close()
- f = io.open(support.TESTFN, "r", encoding="utf-8")
+ f = self.open(support.TESTFN, "r", encoding="utf-8")
s = f.read(prefix_size)
self.assertEquals(s, str(prefix, "ascii"))
self.assertEquals(f.tell(), prefix_size)
self.assertEquals(f.readline(), u_suffix)
- def testSeekingToo(self):
+ def test_seeking_too(self):
# Regression test for a specific bug
data = b'\xe0\xbf\xbf\n'
- f = io.open(support.TESTFN, "wb")
+ f = self.open(support.TESTFN, "wb")
f.write(data)
f.close()
- f = io.open(support.TESTFN, "r", encoding="utf-8")
+ f = self.open(support.TESTFN, "r", encoding="utf-8")
f._CHUNK_SIZE # Just test that it exists
f._CHUNK_SIZE = 2
f.readline()
f.tell()
- def testSeekAndTell(self):
- """Test seek/tell using the StatefulIncrementalDecoder."""
- # Make this test faster by forcing a smaller (but large enough)
- # chunk size. The bigger the chunker size, the slower seek() is,
- # as it tries to replay character decoding one byte at a time.
- CHUNK_SIZE = 256
+ def test_seek_and_tell(self):
+ #Test seek/tell using the StatefulIncrementalDecoder.
+ # Make test faster by doing smaller seeks
+ CHUNK_SIZE = 128
def testSeekAndTellWithData(data, min_pos=0):
"""Tell/seek to various points within a data stream and ensure
that the decoded data returned by read() is consistent."""
- f = io.open(support.TESTFN, 'wb')
+ f = self.open(support.TESTFN, 'wb')
f.write(data)
f.close()
- f = io.open(support.TESTFN, encoding='test_decoder')
+ f = self.open(support.TESTFN, encoding='test_decoder')
+ f._CHUNK_SIZE = CHUNK_SIZE
decoded = f.read()
f.close()
for i in range(min_pos, len(decoded) + 1): # seek positions
for j in [1, 5, len(decoded) - i]: # read lengths
- f = io.open(support.TESTFN, encoding='test_decoder')
- f._CHUNK_SIZE = CHUNK_SIZE
+ f = self.open(support.TESTFN, encoding='test_decoder')
self.assertEquals(f.read(i), decoded[:i])
cookie = f.tell()
self.assertEquals(f.read(j), decoded[i:i + j])
@@ -1079,7 +1690,7 @@
finally:
StatefulIncrementalDecoder.codecEnabled = 0
- def testEncodedWrites(self):
+ def test_encoded_writes(self):
data = "1234567890"
tests = ("utf-16",
"utf-16-le",
@@ -1088,8 +1699,8 @@
"utf-32-le",
"utf-32-be")
for encoding in tests:
- buf = io.BytesIO()
- f = io.TextIOWrapper(buf, encoding=encoding)
+ buf = self.BytesIO()
+ f = self.TextIOWrapper(buf, encoding=encoding)
# Check if the BOM is written only once (see issue1753).
f.write(data)
f.write(data)
@@ -1097,45 +1708,8 @@
self.assertEquals(f.read(), data * 2)
self.assertEquals(buf.getvalue(), (data * 2).encode(encoding))
- def timingTest(self):
- timer = time.time
- enc = "utf8"
- line = "\0\x0f\xff\u0fff\uffff\U000fffff\U0010ffff"*3 + "\n"
- nlines = 10000
- nchars = len(line)
- nbytes = len(line.encode(enc))
- for chunk_size in (32, 64, 128, 256):
- f = io.open(support.TESTFN, "w+", encoding=enc)
- f._CHUNK_SIZE = chunk_size
- t0 = timer()
- for i in range(nlines):
- f.write(line)
- f.flush()
- t1 = timer()
- f.seek(0)
- for line in f:
- pass
- t2 = timer()
- f.seek(0)
- while f.readline():
- pass
- t3 = timer()
- f.seek(0)
- while f.readline():
- f.tell()
- t4 = timer()
- f.close()
- if support.verbose:
- print("\nTiming test: %d lines of %d characters (%d bytes)" %
- (nlines, nchars, nbytes))
- print("File chunk size: %6s" % f._CHUNK_SIZE)
- print("Writing: %6.3f seconds" % (t1-t0))
- print("Reading using iteration: %6.3f seconds" % (t2-t1))
- print("Reading using readline(): %6.3f seconds" % (t3-t2))
- print("Using readline()+tell(): %6.3f seconds" % (t4-t3))
-
- def testReadOneByOne(self):
- txt = io.TextIOWrapper(io.BytesIO(b"AA\r\nBB"))
+ def test_read_one_by_one(self):
+ txt = self.TextIOWrapper(self.BytesIO(b"AA\r\nBB"))
reads = ""
while True:
c = txt.read(1)
@@ -1145,9 +1719,9 @@
self.assertEquals(reads, "AA\nBB")
# read in amounts equal to TextIOWrapper._CHUNK_SIZE which is 128.
- def testReadByChunk(self):
+ def test_read_by_chunk(self):
# make sure "\r\n" straddles 128 char boundary.
- txt = io.TextIOWrapper(io.BytesIO(b"A" * 127 + b"\r\nB"))
+ txt = self.TextIOWrapper(self.BytesIO(b"A" * 127 + b"\r\nB"))
reads = ""
while True:
c = txt.read(128)
@@ -1157,7 +1731,7 @@
self.assertEquals(reads, "A"*127+"\nB")
def test_issue1395_1(self):
- txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ascii")
+ txt = self.TextIOWrapper(self.BytesIO(self.testdata), encoding="ascii")
# read one char at a time
reads = ""
@@ -1169,7 +1743,7 @@
self.assertEquals(reads, self.normalized)
def test_issue1395_2(self):
- txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ascii")
+ txt = self.TextIOWrapper(self.BytesIO(self.testdata), encoding="ascii")
txt._CHUNK_SIZE = 4
reads = ""
@@ -1181,7 +1755,7 @@
self.assertEquals(reads, self.normalized)
def test_issue1395_3(self):
- txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ascii")
+ txt = self.TextIOWrapper(self.BytesIO(self.testdata), encoding="ascii")
txt._CHUNK_SIZE = 4
reads = txt.read(4)
@@ -1192,7 +1766,7 @@
self.assertEquals(reads, self.normalized)
def test_issue1395_4(self):
- txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ascii")
+ txt = self.TextIOWrapper(self.BytesIO(self.testdata), encoding="ascii")
txt._CHUNK_SIZE = 4
reads = txt.read(4)
@@ -1200,7 +1774,7 @@
self.assertEquals(reads, self.normalized)
def test_issue1395_5(self):
- txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ascii")
+ txt = self.TextIOWrapper(self.BytesIO(self.testdata), encoding="ascii")
txt._CHUNK_SIZE = 4
reads = txt.read(4)
@@ -1210,12 +1784,45 @@
self.assertEquals(txt.read(4), "BBB\n")
def test_issue2282(self):
- buffer = io.BytesIO(self.testdata)
- txt = io.TextIOWrapper(buffer, encoding="ascii")
+ buffer = self.BytesIO(self.testdata)
+ txt = self.TextIOWrapper(buffer, encoding="ascii")
self.assertEqual(buffer.seekable(), txt.seekable())
- def check_newline_decoder_utf8(self, decoder):
+class CTextIOWrapperTest(TextIOWrapperTest):
+
+ def test_initialization(self):
+ r = self.BytesIO(b"\xc3\xa9\n\n")
+ b = self.BufferedReader(r, 1000)
+ t = self.TextIOWrapper(b)
+ self.assertRaises(TypeError, t.__init__, b, newline=42)
+ self.assertRaises(ValueError, t.read)
+ self.assertRaises(ValueError, t.__init__, b, newline='xyzzy')
+ self.assertRaises(ValueError, t.read)
+
+ def test_garbage_collection(self):
+ # C TextIOWrapper objects are collected, and collecting them flushes
+ # all data to disk.
+ # The Python version has __del__, so it ends in gc.garbage instead.
+ rawio = io.FileIO(support.TESTFN, "wb")
+ b = self.BufferedWriter(rawio)
+ t = self.TextIOWrapper(b, encoding="ascii")
+ t.write("456def")
+ t.x = t
+ wr = weakref.ref(t)
+ del t
+ gc.collect()
+ self.assert_(wr() is None, wr)
+ with open(support.TESTFN, "rb") as f:
+ self.assertEqual(f.read(), b"456def")
+
+class PyTextIOWrapperTest(TextIOWrapperTest):
+ pass
+
+
+class IncrementalNewlineDecoderTest(unittest.TestCase):
+
+ def check_newline_decoding_utf8(self, decoder):
# UTF-8 specific tests for a newline decoder
def _check_decode(b, s, **kwargs):
# We exercise getstate() / setstate() as well as decode()
@@ -1257,12 +1864,20 @@
_check_decode(b'\xe8\xa2\x88\r', "\u8888")
_check_decode(b'\n', "\n")
- def check_newline_decoder(self, decoder, encoding):
+ def check_newline_decoding(self, decoder, encoding):
result = []
- encoder = codecs.getincrementalencoder(encoding)()
- def _decode_bytewise(s):
- for b in encoder.encode(s):
- result.append(decoder.decode(bytes([b])))
+ if encoding is not None:
+ encoder = codecs.getincrementalencoder(encoding)()
+ def _decode_bytewise(s):
+ # Decode one byte at a time
+ for b in encoder.encode(s):
+ result.append(decoder.decode(bytes([b])))
+ else:
+ encoder = None
+ def _decode_bytewise(s):
+ # Decode one char at a time
+ for c in s:
+ result.append(decoder.decode(c))
self.assertEquals(decoder.newlines, None)
_decode_bytewise("abc\n\r")
self.assertEquals(decoder.newlines, '\n')
@@ -1275,22 +1890,34 @@
_decode_bytewise("abc\r")
self.assertEquals("".join(result), "abc\n\nabcabc\nabcabc")
decoder.reset()
- self.assertEquals(decoder.decode("abc".encode(encoding)), "abc")
+ input = "abc"
+ if encoder is not None:
+ encoder.reset()
+ input = encoder.encode(input)
+ self.assertEquals(decoder.decode(input), "abc")
self.assertEquals(decoder.newlines, None)
def test_newline_decoder(self):
encodings = (
- 'utf-8', 'latin-1',
+ # None meaning the IncrementalNewlineDecoder takes unicode input
+ # rather than bytes input
+ None, 'utf-8', 'latin-1',
'utf-16', 'utf-16-le', 'utf-16-be',
'utf-32', 'utf-32-le', 'utf-32-be',
)
for enc in encodings:
- decoder = codecs.getincrementaldecoder(enc)()
- decoder = io.IncrementalNewlineDecoder(decoder, translate=True)
- self.check_newline_decoder(decoder, enc)
+ decoder = enc and codecs.getincrementaldecoder(enc)()
+ decoder = self.IncrementalNewlineDecoder(decoder, translate=True)
+ self.check_newline_decoding(decoder, enc)
decoder = codecs.getincrementaldecoder("utf-8")()
- decoder = io.IncrementalNewlineDecoder(decoder, translate=True)
- self.check_newline_decoder_utf8(decoder)
+ decoder = self.IncrementalNewlineDecoder(decoder, translate=True)
+ self.check_newline_decoding_utf8(decoder)
+
+class CIncrementalNewlineDecoderTest(IncrementalNewlineDecoderTest):
+ pass
+
+class PyIncrementalNewlineDecoderTest(IncrementalNewlineDecoderTest):
+ pass
# XXX Tests for open()
@@ -1300,24 +1927,23 @@
def tearDown(self):
support.unlink(support.TESTFN)
- def testImport__all__(self):
- for name in io.__all__:
- obj = getattr(io, name, None)
+ def test___all__(self):
+ for name in self.io.__all__:
+ obj = getattr(self.io, name, None)
self.assert_(obj is not None, name)
if name == "open":
continue
elif "error" in name.lower():
self.assert_(issubclass(obj, Exception), name)
else:
- self.assert_(issubclass(obj, io.IOBase))
-
+ self.assert_(issubclass(obj, self.IOBase), name)
def test_attributes(self):
- f = io.open(support.TESTFN, "wb", buffering=0)
+ f = self.open(support.TESTFN, "wb", buffering=0)
self.assertEquals(f.mode, "wb")
f.close()
- f = io.open(support.TESTFN, "U")
+ f = self.open(support.TESTFN, "U")
self.assertEquals(f.name, support.TESTFN)
self.assertEquals(f.buffer.name, support.TESTFN)
self.assertEquals(f.buffer.raw.name, support.TESTFN)
@@ -1326,12 +1952,12 @@
self.assertEquals(f.buffer.raw.mode, "rb")
f.close()
- f = io.open(support.TESTFN, "w+")
+ f = self.open(support.TESTFN, "w+")
self.assertEquals(f.mode, "w+")
self.assertEquals(f.buffer.mode, "rb+") # Does it really matter?
self.assertEquals(f.buffer.raw.mode, "rb+")
- g = io.open(f.fileno(), "wb", closefd=False)
+ g = self.open(f.fileno(), "wb", closefd=False)
self.assertEquals(g.mode, "wb")
self.assertEquals(g.raw.mode, "wb")
self.assertEquals(g.name, f.fileno())
@@ -1357,13 +1983,12 @@
{"mode": "w+", "buffering": 2},
{"mode": "w+b", "buffering": 0},
]:
- f = io.open(support.TESTFN, **kwargs)
+ f = self.open(support.TESTFN, **kwargs)
f.close()
self.assertRaises(ValueError, f.flush)
self.assertRaises(ValueError, f.fileno)
self.assertRaises(ValueError, f.isatty)
self.assertRaises(ValueError, f.__iter__)
- self.assertRaises(ValueError, next, f)
if hasattr(f, "peek"):
self.assertRaises(ValueError, f.peek, 1)
self.assertRaises(ValueError, f.read)
@@ -1376,16 +2001,101 @@
self.assertRaises(ValueError, f.seek, 0)
self.assertRaises(ValueError, f.tell)
self.assertRaises(ValueError, f.truncate)
- self.assertRaises(ValueError, f.write, "")
+ self.assertRaises(ValueError, f.write,
+ b"" if "b" in kwargs['mode'] else "")
self.assertRaises(ValueError, f.writelines, [])
+ self.assertRaises(ValueError, next, f)
+ def test_blockingioerror(self):
+ # Various BlockingIOError issues
+ self.assertRaises(TypeError, self.BlockingIOError)
+ self.assertRaises(TypeError, self.BlockingIOError, 1)
+ self.assertRaises(TypeError, self.BlockingIOError, 1, 2, 3, 4)
+ self.assertRaises(TypeError, self.BlockingIOError, 1, "", None)
+ b = self.BlockingIOError(1, "")
+ self.assertEqual(b.characters_written, 0)
+ class C(str):
+ pass
+ c = C("")
+ b = self.BlockingIOError(1, c)
+ c.b = b
+ b.c = c
+ wr = weakref.ref(c)
+ del c, b
+ gc.collect()
+ self.assert_(wr() is None, wr)
+
+ def test_abcs(self):
+ # Test the visible base classes are ABCs.
+ self.assertTrue(isinstance(self.IOBase, abc.ABCMeta))
+ self.assertTrue(isinstance(self.RawIOBase, abc.ABCMeta))
+ self.assertTrue(isinstance(self.BufferedIOBase, abc.ABCMeta))
+ self.assertTrue(isinstance(self.TextIOBase, abc.ABCMeta))
+
+ def _check_abc_inheritance(self, abcmodule):
+ with self.open(support.TESTFN, "wb", buffering=0) as f:
+ self.assertTrue(isinstance(f, abcmodule.IOBase))
+ self.assertTrue(isinstance(f, abcmodule.RawIOBase))
+ self.assertFalse(isinstance(f, abcmodule.BufferedIOBase))
+ self.assertFalse(isinstance(f, abcmodule.TextIOBase))
+ with self.open(support.TESTFN, "wb") as f:
+ self.assertTrue(isinstance(f, abcmodule.IOBase))
+ self.assertFalse(isinstance(f, abcmodule.RawIOBase))
+ self.assertTrue(isinstance(f, abcmodule.BufferedIOBase))
+ self.assertFalse(isinstance(f, abcmodule.TextIOBase))
+ with self.open(support.TESTFN, "w") as f:
+ self.assertTrue(isinstance(f, abcmodule.IOBase))
+ self.assertFalse(isinstance(f, abcmodule.RawIOBase))
+ self.assertFalse(isinstance(f, abcmodule.BufferedIOBase))
+ self.assertTrue(isinstance(f, abcmodule.TextIOBase))
+
+ def test_abc_inheritance(self):
+ # Test implementations inherit from their respective ABCs
+ self._check_abc_inheritance(self)
+
+ def test_abc_inheritance_official(self):
+ # Test implementations inherit from the official ABCs of the
+ # baseline "io" module.
+ self._check_abc_inheritance(io)
+
+class CMiscIOTest(MiscIOTest):
+ io = io
+
+class PyMiscIOTest(MiscIOTest):
+ io = pyio
def test_main():
- support.run_unittest(IOTest, BytesIOTest, StringIOTest,
- BufferedReaderTest, BufferedWriterTest,
- BufferedRWPairTest, BufferedRandomTest,
- StatefulIncrementalDecoderTest,
- TextIOWrapperTest, MiscIOTest)
+ tests = (CIOTest, PyIOTest,
+ CBufferedReaderTest, PyBufferedReaderTest,
+ CBufferedWriterTest, PyBufferedWriterTest,
+ CBufferedRWPairTest, PyBufferedRWPairTest,
+ CBufferedRandomTest, PyBufferedRandomTest,
+ StatefulIncrementalDecoderTest,
+ CIncrementalNewlineDecoderTest, PyIncrementalNewlineDecoderTest,
+ CTextIOWrapperTest, PyTextIOWrapperTest,
+ CMiscIOTest, PyMiscIOTest,)
+
+ # Put the namespaces of the IO module we are testing and some useful mock
+ # classes in the __dict__ of each test.
+ mocks = (MockRawIO, MisbehavedRawIO, MockFileIO, CloseFailureIO,
+ MockNonBlockWriterIO)
+ all_members = io.__all__ + ["IncrementalNewlineDecoder"]
+ c_io_ns = {name : getattr(io, name) for name in all_members}
+ py_io_ns = {name : getattr(pyio, name) for name in all_members}
+ globs = globals()
+ c_io_ns.update((x.__name__, globs["C" + x.__name__]) for x in mocks)
+ py_io_ns.update((x.__name__, globs["Py" + x.__name__]) for x in mocks)
+ # Avoid turning open into a bound method.
+ py_io_ns["open"] = pyio.OpenWrapper
+ for test in tests:
+ if test.__name__.startswith("C"):
+ for name, obj in c_io_ns.items():
+ setattr(test, name, obj)
+ elif test.__name__.startswith("Py"):
+ for name, obj in py_io_ns.items():
+ setattr(test, name, obj)
+
+ support.run_unittest(*tests)
if __name__ == "__main__":
- unittest.main()
+ test_main()
Modified: python/branches/py3k/Lib/test/test_largefile.py
==============================================================================
--- python/branches/py3k/Lib/test/test_largefile.py (original)
+++ python/branches/py3k/Lib/test/test_largefile.py Wed Mar 4 01:14:51 2009
@@ -7,6 +7,8 @@
import unittest
from test.support import run_unittest, TESTFN, verbose, requires, \
TestSkipped, unlink
+import io # C implementation of io
+import _pyio as pyio # Python implementation of io
try:
import signal
@@ -21,7 +23,7 @@
size = 2500000000
-class TestCase(unittest.TestCase):
+class LargeFileTest(unittest.TestCase):
"""Test that each file function works as expected for a large
(i.e. > 2GB, do we have to check > 4GB) files.
@@ -34,7 +36,7 @@
def test_seek(self):
if verbose:
print('create large file via seek (may be sparse file) ...')
- with open(TESTFN, 'wb') as f:
+ with self.open(TESTFN, 'wb') as f:
f.write(b'z')
f.seek(0)
f.seek(size)
@@ -52,7 +54,7 @@
def test_seek_read(self):
if verbose:
print('play around with seek() and read() with the built largefile')
- with open(TESTFN, 'rb') as f:
+ with self.open(TESTFN, 'rb') as f:
self.assertEqual(f.tell(), 0)
self.assertEqual(f.read(1), b'z')
self.assertEqual(f.tell(), 1)
@@ -85,7 +87,7 @@
def test_lseek(self):
if verbose:
print('play around with os.lseek() with the built largefile')
- with open(TESTFN, 'rb') as f:
+ with self.open(TESTFN, 'rb') as f:
self.assertEqual(os.lseek(f.fileno(), 0, 0), 0)
self.assertEqual(os.lseek(f.fileno(), 42, 0), 42)
self.assertEqual(os.lseek(f.fileno(), 42, 1), 84)
@@ -100,7 +102,7 @@
def test_truncate(self):
if verbose:
print('try truncate')
- with open(TESTFN, 'r+b') as f:
+ with self.open(TESTFN, 'r+b') as f:
# this is already decided before start running the test suite
# but we do it anyway for extra protection
if not hasattr(f, 'truncate'):
@@ -143,7 +145,7 @@
# Only run if the current filesystem supports large files.
# (Skip this test on Windows, since we now always support
# large files.)
- f = open(TESTFN, 'wb')
+ f = open(TESTFN, 'wb', buffering=0)
try:
# 2**31 == 2147483648
f.seek(2147483649)
@@ -158,14 +160,19 @@
else:
f.close()
suite = unittest.TestSuite()
- suite.addTest(TestCase('test_seek'))
- suite.addTest(TestCase('test_osstat'))
- suite.addTest(TestCase('test_seek_read'))
- suite.addTest(TestCase('test_lseek'))
- with open(TESTFN, 'w') as f:
- if hasattr(f, 'truncate'):
- suite.addTest(TestCase('test_truncate'))
- unlink(TESTFN)
+ for _open, prefix in [(io.open, 'C'), (pyio.open, 'Py')]:
+ class TestCase(LargeFileTest):
+ pass
+ TestCase.open = staticmethod(_open)
+ TestCase.__name__ = prefix + LargeFileTest.__name__
+ suite.addTest(TestCase('test_seek'))
+ suite.addTest(TestCase('test_osstat'))
+ suite.addTest(TestCase('test_seek_read'))
+ suite.addTest(TestCase('test_lseek'))
+ with _open(TESTFN, 'wb') as f:
+ if hasattr(f, 'truncate'):
+ suite.addTest(TestCase('test_truncate'))
+ unlink(TESTFN)
try:
run_unittest(suite)
finally:
Modified: python/branches/py3k/Lib/test/test_memoryio.py
==============================================================================
--- python/branches/py3k/Lib/test/test_memoryio.py (original)
+++ python/branches/py3k/Lib/test/test_memoryio.py Wed Mar 4 01:14:51 2009
@@ -7,13 +7,52 @@
from test import support
import io
+import _pyio as pyio
import sys
-try:
- import _bytesio, _stringio
- has_c_implementation = True
-except ImportError:
- has_c_implementation = False
+class MemorySeekTestMixin:
+
+ def testInit(self):
+ buf = self.buftype("1234567890")
+ bytesIo = self.ioclass(buf)
+
+ def testRead(self):
+ buf = self.buftype("1234567890")
+ bytesIo = self.ioclass(buf)
+
+ self.assertEquals(buf[:1], bytesIo.read(1))
+ self.assertEquals(buf[1:5], bytesIo.read(4))
+ self.assertEquals(buf[5:], bytesIo.read(900))
+ self.assertEquals(self.EOF, bytesIo.read())
+
+ def testReadNoArgs(self):
+ buf = self.buftype("1234567890")
+ bytesIo = self.ioclass(buf)
+
+ self.assertEquals(buf, bytesIo.read())
+ self.assertEquals(self.EOF, bytesIo.read())
+
+ def testSeek(self):
+ buf = self.buftype("1234567890")
+ bytesIo = self.ioclass(buf)
+
+ bytesIo.read(5)
+ bytesIo.seek(0)
+ self.assertEquals(buf, bytesIo.read())
+
+ bytesIo.seek(3)
+ self.assertEquals(buf[3:], bytesIo.read())
+ self.assertRaises(TypeError, bytesIo.seek, 0.0)
+
+ def testTell(self):
+ buf = self.buftype("1234567890")
+ bytesIo = self.ioclass(buf)
+
+ self.assertEquals(0, bytesIo.tell())
+ bytesIo.seek(5)
+ self.assertEquals(5, bytesIo.tell())
+ bytesIo.seek(10000)
+ self.assertEquals(10000, bytesIo.tell())
class MemoryTestMixin:
@@ -148,7 +187,7 @@
self.assertEqual(memio.readline(), self.EOF)
memio.seek(0)
self.assertEqual(type(memio.readline()), type(buf))
- self.assertEqual(memio.readline(None), buf)
+ self.assertEqual(memio.readline(), buf)
self.assertRaises(TypeError, memio.readline, '')
memio.close()
self.assertRaises(ValueError, memio.readline)
@@ -296,11 +335,11 @@
self.assertEqual(test2(), buf)
-class PyBytesIOTest(MemoryTestMixin, unittest.TestCase):
+class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
@staticmethod
def buftype(s):
return s.encode("ascii")
- ioclass = io._BytesIO
+ ioclass = pyio.BytesIO
EOF = b""
def test_read1(self):
@@ -371,11 +410,32 @@
self.assertEqual(memio.getvalue(), buf)
-class PyStringIOTest(MemoryTestMixin, unittest.TestCase):
+class PyStringIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
buftype = str
- ioclass = io._StringIO
+ ioclass = pyio.StringIO
EOF = ""
+ # TextIO-specific behaviour.
+
+ def test_newlines_property(self):
+ memio = self.ioclass(newline=None)
+ # The C StringIO decodes newlines in write() calls, but the Python
+ # implementation only does when reading. This function forces them to
+ # be decoded for testing.
+ def force_decode():
+ memio.seek(0)
+ memio.read()
+ self.assertEqual(memio.newlines, None)
+ memio.write("a\n")
+ force_decode()
+ self.assertEqual(memio.newlines, "\n")
+ memio.write("b\r\n")
+ force_decode()
+ self.assertEqual(memio.newlines, ("\n", "\r\n"))
+ memio.write("c\rd")
+ force_decode()
+ self.assertEqual(memio.newlines, ("\r", "\n", "\r\n"))
+
def test_relative_seek(self):
memio = self.ioclass()
@@ -386,32 +446,99 @@
self.assertRaises(IOError, memio.seek, 1, 1)
self.assertRaises(IOError, memio.seek, 1, 2)
+ def test_textio_properties(self):
+ memio = self.ioclass()
+
+ # These are just dummy values but we nevertheless check them for fear
+ # of unexpected breakage.
+ self.assertEqual(memio.encoding, "utf-8")
+ self.assertEqual(memio.errors, "strict")
+ self.assertEqual(memio.line_buffering, False)
+
+ def test_newline_none(self):
+ # newline=None
+ memio = self.ioclass("a\nb\r\nc\rd", newline=None)
+ self.assertEqual(list(memio), ["a\n", "b\n", "c\n", "d"])
+ memio.seek(0)
+ self.assertEqual(memio.read(1), "a")
+ self.assertEqual(memio.read(2), "\nb")
+ self.assertEqual(memio.read(2), "\nc")
+ self.assertEqual(memio.read(1), "\n")
+ memio = self.ioclass(newline=None)
+ self.assertEqual(2, memio.write("a\n"))
+ self.assertEqual(3, memio.write("b\r\n"))
+ self.assertEqual(3, memio.write("c\rd"))
+ memio.seek(0)
+ self.assertEqual(memio.read(), "a\nb\nc\nd")
+ memio = self.ioclass("a\r\nb", newline=None)
+ self.assertEqual(memio.read(3), "a\nb")
+
+ def test_newline_empty(self):
+ # newline=""
+ memio = self.ioclass("a\nb\r\nc\rd", newline="")
+ self.assertEqual(list(memio), ["a\n", "b\r\n", "c\r", "d"])
+ memio.seek(0)
+ self.assertEqual(memio.read(4), "a\nb\r")
+ self.assertEqual(memio.read(2), "\nc")
+ self.assertEqual(memio.read(1), "\r")
+ memio = self.ioclass(newline="")
+ self.assertEqual(2, memio.write("a\n"))
+ self.assertEqual(2, memio.write("b\r"))
+ self.assertEqual(2, memio.write("\nc"))
+ self.assertEqual(2, memio.write("\rd"))
+ memio.seek(0)
+ self.assertEqual(list(memio), ["a\n", "b\r\n", "c\r", "d"])
+
+ def test_newline_lf(self):
+ # newline="\n"
+ memio = self.ioclass("a\nb\r\nc\rd")
+ self.assertEqual(list(memio), ["a\n", "b\r\n", "c\rd"])
+
+ def test_newline_cr(self):
+ # newline="\r"
+ memio = self.ioclass("a\nb\r\nc\rd", newline="\r")
+ memio.seek(0)
+ self.assertEqual(memio.read(), "a\rb\r\rc\rd")
+ memio.seek(0)
+ self.assertEqual(list(memio), ["a\r", "b\r", "\r", "c\r", "d"])
+
+ def test_newline_crlf(self):
+ # newline="\r\n"
+ memio = self.ioclass("a\nb\r\nc\rd", newline="\r\n")
+ memio.seek(0)
+ self.assertEqual(memio.read(), "a\r\nb\r\r\nc\rd")
+ memio.seek(0)
+ self.assertEqual(list(memio), ["a\r\n", "b\r\r\n", "c\rd"])
+
+ def test_issue5265(self):
+ # StringIO can duplicate newlines in universal newlines mode
+ memio = self.ioclass("a\r\nb\r\n", newline=None)
+ self.assertEqual(memio.read(5), "a\nb\n")
+
+
+class CBytesIOTest(PyBytesIOTest):
+ ioclass = io.BytesIO
+
+class CStringIOTest(PyStringIOTest):
+ ioclass = io.StringIO
+
# XXX: For the Python version of io.StringIO, this is highly
# dependent on the encoding used for the underlying buffer.
- # def test_widechar(self):
- # buf = self.buftype("\U0002030a\U00020347")
- # memio = self.ioclass(buf)
- #
- # self.assertEqual(memio.getvalue(), buf)
- # self.assertEqual(memio.write(buf), len(buf))
- # self.assertEqual(memio.tell(), len(buf))
- # self.assertEqual(memio.getvalue(), buf)
- # self.assertEqual(memio.write(buf), len(buf))
- # self.assertEqual(memio.tell(), len(buf) * 2)
- # self.assertEqual(memio.getvalue(), buf + buf)
-
-if has_c_implementation:
- class CBytesIOTest(PyBytesIOTest):
- ioclass = io.BytesIO
+ def test_widechar(self):
+ buf = self.buftype("\U0002030a\U00020347")
+ memio = self.ioclass(buf)
- class CStringIOTest(PyStringIOTest):
- ioclass = io.StringIO
+ self.assertEqual(memio.getvalue(), buf)
+ self.assertEqual(memio.write(buf), len(buf))
+ self.assertEqual(memio.tell(), len(buf))
+ self.assertEqual(memio.getvalue(), buf)
+ self.assertEqual(memio.write(buf), len(buf))
+ self.assertEqual(memio.tell(), len(buf) * 2)
+ self.assertEqual(memio.getvalue(), buf + buf)
def test_main():
- tests = [PyBytesIOTest, PyStringIOTest]
- if has_c_implementation:
- tests.extend([CBytesIOTest, CStringIOTest])
+ tests = [PyBytesIOTest, PyStringIOTest, CBytesIOTest, CStringIOTest]
support.run_unittest(*tests)
if __name__ == '__main__':
Modified: python/branches/py3k/Lib/test/test_univnewlines.py
==============================================================================
--- python/branches/py3k/Lib/test/test_univnewlines.py (original)
+++ python/branches/py3k/Lib/test/test_univnewlines.py Wed Mar 4 01:14:51 2009
@@ -1,4 +1,6 @@
# Tests universal newline support for both reading and parsing files.
+import io
+import _pyio as pyio
import unittest
import os
import sys
@@ -35,7 +37,7 @@
WRITEMODE = 'wb'
def setUp(self):
- fp = open(support.TESTFN, self.WRITEMODE)
+ fp = self.open(support.TESTFN, self.WRITEMODE)
data = self.DATA
if "b" in self.WRITEMODE:
data = data.encode("ascii")
@@ -49,19 +51,19 @@
pass
def test_read(self):
- fp = open(support.TESTFN, self.READMODE)
+ fp = self.open(support.TESTFN, self.READMODE)
data = fp.read()
self.assertEqual(data, DATA_LF)
self.assertEqual(repr(fp.newlines), repr(self.NEWLINE))
def test_readlines(self):
- fp = open(support.TESTFN, self.READMODE)
+ fp = self.open(support.TESTFN, self.READMODE)
data = fp.readlines()
self.assertEqual(data, DATA_SPLIT)
self.assertEqual(repr(fp.newlines), repr(self.NEWLINE))
def test_readline(self):
- fp = open(support.TESTFN, self.READMODE)
+ fp = self.open(support.TESTFN, self.READMODE)
data = []
d = fp.readline()
while d:
@@ -71,7 +73,7 @@
self.assertEqual(repr(fp.newlines), repr(self.NEWLINE))
def test_seek(self):
- fp = open(support.TESTFN, self.READMODE)
+ fp = self.open(support.TESTFN, self.READMODE)
fp.readline()
pos = fp.tell()
data = fp.readlines()
@@ -94,7 +96,7 @@
DATA = DATA_CRLF
def test_tell(self):
- fp = open(support.TESTFN, self.READMODE)
+ fp = self.open(support.TESTFN, self.READMODE)
self.assertEqual(repr(fp.newlines), repr(None))
data = fp.readline()
pos = fp.tell()
@@ -106,12 +108,22 @@
def test_main():
- support.run_unittest(
- TestCRNewlines,
- TestLFNewlines,
- TestCRLFNewlines,
- TestMixedNewlines
- )
+ base_tests = (TestCRNewlines,
+ TestLFNewlines,
+ TestCRLFNewlines,
+ TestMixedNewlines)
+ tests = []
+ # Test the C and Python implementations.
+ for test in base_tests:
+ class CTest(test):
+ open = io.open
+ CTest.__name__ = "C" + test.__name__
+ class PyTest(test):
+ open = staticmethod(pyio.open)
+ PyTest.__name__ = "Py" + test.__name__
+ tests.append(CTest)
+ tests.append(PyTest)
+ support.run_unittest(*tests)
if __name__ == '__main__':
test_main()
Modified: python/branches/py3k/Lib/test/test_uu.py
==============================================================================
--- python/branches/py3k/Lib/test/test_uu.py (original)
+++ python/branches/py3k/Lib/test/test_uu.py Wed Mar 4 01:14:51 2009
@@ -32,6 +32,8 @@
encoding=encoding,
errors=errors,
newline=newline)
+ self._encoding = encoding
+ self._errors = errors
if initial_value:
if not isinstance(initial_value, str):
initial_value = str(initial_value)
Modified: python/branches/py3k/Makefile.pre.in
==============================================================================
--- python/branches/py3k/Makefile.pre.in (original)
+++ python/branches/py3k/Makefile.pre.in Wed Mar 4 01:14:51 2009
@@ -193,6 +193,15 @@
# Used of signalmodule.o is not available
SIGNAL_OBJS= @SIGNAL_OBJS@
+IO_H= Modules/_iomodule.h
+
+IO_OBJS= \
+ Modules/io.o \
+ Modules/_iobase.o \
+ Modules/_fileio.o \
+ Modules/_bufferedio.o \
+ Modules/_textio.o \
+ Modules/_bytesio.o
##########################################################################
# Grammar
@@ -521,6 +530,7 @@
Modules/python.o: $(srcdir)/Modules/python.c
$(MAINCC) -c $(PY_CFLAGS) -o $@ $(srcdir)/Modules/python.c
+$(IO_OBJS): $(IO_H)
$(GRAMMAR_H) $(GRAMMAR_C): $(PGEN) $(GRAMMAR_INPUT)
-@$(INSTALL) -d Include
Modified: python/branches/py3k/Misc/NEWS
==============================================================================
--- python/branches/py3k/Misc/NEWS (original)
+++ python/branches/py3k/Misc/NEWS Wed Mar 4 01:14:51 2009
@@ -14,6 +14,8 @@
=======
+- The io module has been reimplemented in C for speed.
+
- Give dict views an informative __repr__.
- Issue #5247: Improve error message when unknown format codes are
Modified: python/branches/py3k/Modules/Setup.dist
==============================================================================
--- python/branches/py3k/Modules/Setup.dist (original)
+++ python/branches/py3k/Modules/Setup.dist Wed Mar 4 01:14:51 2009
@@ -111,10 +111,10 @@
# if $HOME is not set
_sre _sre.c # Fredrik Lundh's new regular expressions
_codecs _codecsmodule.c # access to the builtin codecs and codec registry
-_fileio _fileio.c # Standard I/O baseline
_weakref _weakref.c # weak references
-_bytesio _bytesio.c # For Lib/io.py
-_stringio _stringio.c # For Lib/io.py
+
+# Standard I/O baseline
+_io io.c _iobase.c _fileio.c _bytesio.c _bufferedio.c _textio.c _stringio.c
# The zipimport module is always imported at startup. Having it as a
# builtin module avoids some bootstrapping problems and reduces overhead.
@@ -164,7 +164,6 @@
#_collections _collectionsmodule.c # Container types
#itertools itertoolsmodule.c # Functions creating iterators for efficient looping
#atexit atexitmodule.c # Register functions to be run at interpreter-shutdown
-#_functools _functoolsmodule.c # Tools for working with functions and callable objects
#_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _elementtree.c # elementtree accelerator
#_pickle _pickle.c # pickle accelerator
#datetime datetimemodule.c # date/time type
Copied: python/branches/py3k/Modules/_bufferedio.c (from r68756, /python/branches/io-c/Modules/_bufferedio.c)
==============================================================================
--- /python/branches/io-c/Modules/_bufferedio.c (original)
+++ python/branches/py3k/Modules/_bufferedio.c Wed Mar 4 01:14:51 2009
@@ -1,3 +1,12 @@
+/*
+ An implementation of Buffered I/O as defined by PEP 3116 - "New I/O"
+
+ Classes defined here: BufferedIOBase, BufferedReader, BufferedWriter,
+ BufferedRandom.
+
+ Written by Amaury Forgeot d'Arc and Antoine Pitrou
+*/
+
#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
@@ -60,7 +69,7 @@
static PyObject *
BufferedIOBase_unsupported(const char *message)
{
- PyErr_SetString(PyIOExc_UnsupportedOperation, message);
+ PyErr_SetString(IO_STATE->unsupported_operation, message);
return NULL;
}
@@ -127,7 +136,7 @@
PyTypeObject PyBufferedIOBase_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "BufferedIOBase", /*tp_name*/
+ "_io._BufferedIOBase", /*tp_name*/
0, /*tp_basicsize*/
0, /*tp_itemsize*/
0, /*tp_dealloc*/
@@ -285,8 +294,9 @@
static void
BufferedObject_dealloc(BufferedObject *self)
{
- if (_PyIOBase_finalize((PyObject *) self) < 0)
+ if (self->ok && _PyIOBase_finalize((PyObject *) self) < 0)
return;
+ _PyObject_GC_UNTRACK(self);
self->ok = 0;
if (self->weakreflist != NULL)
PyObject_ClearWeakRefs((PyObject *)self);
@@ -303,6 +313,24 @@
Py_TYPE(self)->tp_free((PyObject *)self);
}
+static int
+Buffered_traverse(BufferedObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->raw);
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+Buffered_clear(BufferedObject *self)
+{
+ if (self->ok && _PyIOBase_finalize((PyObject *) self) < 0)
+ return -1;
+ self->ok = 0;
+ Py_CLEAR(self->raw);
+ Py_CLEAR(self->dict);
+ return 0;
+}
/*
* _BufferedIOMixin methods
@@ -310,45 +338,6 @@
* by BufferedReader and BufferedWriter
*/
-/* Positioning */
-
-static PyObject *
-BufferedIOMixin_truncate(BufferedObject *self, PyObject *args)
-{
- PyObject *pos = Py_None;
- PyObject *res;
-
- CHECK_INITIALIZED(self)
- if (!PyArg_ParseTuple(args, "|O:truncate", &pos)) {
- return NULL;
- }
-
- /* Flush the stream. We're mixing buffered I/O with lower-level I/O,
- * and a flush may be necessary to synch both views of the current
- * file state.
- */
- res = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_flush, NULL);
- if (res == NULL)
- return NULL;
- Py_DECREF(res);
-
- if (pos == Py_None) {
- pos = PyObject_CallMethod(self->raw, "tell", NULL);
- if (pos == NULL)
- return NULL;
- }
- else
- Py_INCREF(pos);
-
- /* XXX: Should seek() be used, instead of passing the position
- * XXX directly to truncate?
- */
- res = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_truncate, pos, NULL);
- Py_DECREF(pos);
-
- return res;
-}
-
/* Flush and close */
static PyObject *
@@ -715,7 +704,7 @@
all `n` bytes asked by the caller (and possibly more, so as to fill
our buffer for the next reads). */
- have = READAHEAD(self);
+ have = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
if (have > 0) {
if (n > have)
n = have;
@@ -783,7 +772,7 @@
ENTER_BUFFERED(self)
/* First, try to find a line in the buffer */
- n = READAHEAD(self);
+ n = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
if (limit >= 0 && n > limit)
n = limit;
start = self->buffer + self->pos;
@@ -871,7 +860,7 @@
goto end;
}
Py_CLEAR(res);
- res = _PyBytes_Join(PyBytes_FromStringAndSize(NULL, 0), chunks);
+ res = _PyBytes_Join(_PyIO_empty_bytes, chunks);
end:
LEAVE_BUFFERED(self)
@@ -987,9 +976,6 @@
return NULL;
}
- if (pos != Py_None)
- Py_INCREF(pos);
-
ENTER_BUFFERED(self)
if (self->writable) {
@@ -1140,7 +1126,7 @@
{
Py_ssize_t start, len, n;
if (VALID_READ_BUFFER(self))
- start = self->read_end;
+ start = Py_SAFE_DOWNCAST(self->read_end, Py_off_t, Py_ssize_t);
else
start = 0;
len = self->buffer_size - start;
@@ -1167,7 +1153,7 @@
return NULL;
/* First copy what we have in the current buffer. */
- current_size = READAHEAD(self);
+ current_size = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
data = NULL;
if (current_size) {
data = PyBytes_FromStringAndSize(
@@ -1236,7 +1222,7 @@
}
/* The number of bytes to read is specified, return at most n bytes. */
- current_size = READAHEAD(self);
+ current_size = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
if (n <= current_size) {
/* Fast path: the data to read is fully buffered. */
res = PyBytes_FromStringAndSize(self->buffer + self->pos, n);
@@ -1331,7 +1317,7 @@
{
Py_ssize_t have, r;
- have = READAHEAD(self);
+ have = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
/* Constraints:
1. we don't want to advance the file position.
2. we don't want to lose block alignment, so we can't shift the buffer
@@ -1355,7 +1341,6 @@
static PyMethodDef BufferedReader_methods[] = {
/* BufferedIOMixin methods */
- {"truncate", (PyCFunction)BufferedIOMixin_truncate, METH_VARARGS},
{"flush", (PyCFunction)BufferedIOMixin_flush, METH_NOARGS},
{"close", (PyCFunction)BufferedIOMixin_close, METH_NOARGS},
{"seekable", (PyCFunction)BufferedIOMixin_seekable, METH_NOARGS},
@@ -1370,6 +1355,7 @@
{"readline", (PyCFunction)Buffered_readline, METH_VARARGS},
{"seek", (PyCFunction)Buffered_seek, METH_VARARGS},
{"tell", (PyCFunction)Buffered_tell, METH_NOARGS},
+ {"truncate", (PyCFunction)Buffered_truncate, METH_VARARGS},
{NULL, NULL}
};
@@ -1388,7 +1374,7 @@
PyTypeObject PyBufferedReader_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "BufferedReader", /*tp_name*/
+ "_io.BufferedReader", /*tp_name*/
sizeof(BufferedObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)BufferedObject_dealloc, /*tp_dealloc*/
@@ -1406,12 +1392,13 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
BufferedReader_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)Buffered_traverse, /* tp_traverse */
+ (inquiry)Buffered_clear, /* tp_clear */
0, /* tp_richcompare */
- offsetof(BufferedObject, weakreflist), /*tp_weaklistoffset*/
+ offsetof(BufferedObject, weakreflist), /*tp_weaklistoffset*/
0, /* tp_iter */
(iternextfunc)Buffered_iternext, /* tp_iternext */
BufferedReader_methods, /* tp_methods */
@@ -1532,7 +1519,8 @@
while (self->write_pos < self->write_end) {
n = _BufferedWriter_raw_write(self,
self->buffer + self->write_pos,
- self->write_end - self->write_pos);
+ Py_SAFE_DOWNCAST(self->write_end - self->write_pos,
+ Py_off_t, Py_ssize_t));
if (n == -1) {
Py_ssize_t *w = _Buffered_check_blocking_error();
if (w == NULL)
@@ -1546,7 +1534,7 @@
}
self->write_pos += n;
self->raw_pos = self->write_pos;
- written += n;
+ written += Py_SAFE_DOWNCAST(n, Py_off_t, Py_ssize_t);
}
if (restore_pos) {
@@ -1593,7 +1581,7 @@
self->pos = 0;
self->raw_pos = 0;
}
- avail = self->buffer_size - self->pos;
+ avail = Py_SAFE_DOWNCAST(self->buffer_size - self->pos, Py_off_t, Py_ssize_t);
if (buf.len <= avail) {
memcpy(self->buffer + self->pos, buf.buf, buf.len);
if (!VALID_WRITE_BUFFER(self)) {
@@ -1617,12 +1605,14 @@
/* Make some place by shifting the buffer. */
assert(VALID_WRITE_BUFFER(self));
memmove(self->buffer, self->buffer + self->write_pos,
- self->write_end - self->write_pos);
+ Py_SAFE_DOWNCAST(self->write_end - self->write_pos,
+ Py_off_t, Py_ssize_t));
self->write_end -= self->write_pos;
self->raw_pos -= self->write_pos;
self->pos -= self->write_pos;
self->write_pos = 0;
- avail = self->buffer_size - self->write_end;
+ avail = Py_SAFE_DOWNCAST(self->buffer_size - self->write_end,
+ Py_off_t, Py_ssize_t);
if (buf.len <= avail) {
/* Everything can be buffered */
PyErr_Clear();
@@ -1722,7 +1712,7 @@
PyTypeObject PyBufferedWriter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "BufferedWriter", /*tp_name*/
+ "_io.BufferedWriter", /*tp_name*/
sizeof(BufferedObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)BufferedObject_dealloc, /*tp_dealloc*/
@@ -1740,10 +1730,11 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
BufferedWriter_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)Buffered_traverse, /* tp_traverse */
+ (inquiry)Buffered_clear, /* tp_clear */
0, /* tp_richcompare */
offsetof(BufferedObject, weakreflist), /*tp_weaklistoffset*/
0, /* tp_iter */
@@ -1789,6 +1780,7 @@
BufferedObject *reader;
BufferedObject *writer;
PyObject *dict;
+ PyObject *weakreflist;
} BufferedRWPairObject;
static int
@@ -1835,11 +1827,30 @@
return 0;
}
+static int
+BufferedRWPair_traverse(BufferedRWPairObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+BufferedRWPair_clear(BufferedRWPairObject *self)
+{
+ Py_CLEAR(self->reader);
+ Py_CLEAR(self->writer);
+ Py_CLEAR(self->dict);
+ return 0;
+}
+
static void
BufferedRWPair_dealloc(BufferedRWPairObject *self)
{
+ _PyObject_GC_UNTRACK(self);
Py_CLEAR(self->reader);
Py_CLEAR(self->writer);
+ Py_CLEAR(self->dict);
+ Py_TYPE(self)->tp_free((PyObject *) self);
}
static PyObject *
@@ -1946,8 +1957,8 @@
PyTypeObject PyBufferedRWPair_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "BufferedRWPair", /*tp_name*/
- sizeof(BufferedRWPairObject), /*tp_basicsize*/
+ "_io.BufferedRWPair", /*tp_name*/
+ sizeof(BufferedRWPairObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)BufferedRWPair_dealloc, /*tp_dealloc*/
0, /*tp_print*/
@@ -1964,17 +1975,18 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /* tp_flags */
BufferedRWPair_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)BufferedRWPair_traverse, /* tp_traverse */
+ (inquiry)BufferedRWPair_clear, /* tp_clear */
0, /* tp_richcompare */
- offsetof(BufferedObject, weakreflist), /*tp_weaklistoffset*/
+ offsetof(BufferedRWPairObject, weakreflist), /*tp_weaklistoffset*/
0, /* tp_iter */
0, /* tp_iternext */
BufferedRWPair_methods, /* tp_methods */
- 0, /* tp_members */
- 0, /* tp_getset */
+ 0, /* tp_members */
+ 0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
@@ -2076,7 +2088,7 @@
PyTypeObject PyBufferedRandom_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "BufferedRandom", /*tp_name*/
+ "_io.BufferedRandom", /*tp_name*/
sizeof(BufferedObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)BufferedObject_dealloc, /*tp_dealloc*/
@@ -2094,10 +2106,11 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
BufferedRandom_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)Buffered_traverse, /* tp_traverse */
+ (inquiry)Buffered_clear, /* tp_clear */
0, /* tp_richcompare */
offsetof(BufferedObject, weakreflist), /*tp_weaklistoffset*/
0, /* tp_iter */
Modified: python/branches/py3k/Modules/_bytesio.c
==============================================================================
--- python/branches/py3k/Modules/_bytesio.c (original)
+++ python/branches/py3k/Modules/_bytesio.c Wed Mar 4 01:14:51 2009
@@ -1,4 +1,6 @@
#include "Python.h"
+#include "structmember.h" /* for offsetof() */
+#include "_iomodule.h"
typedef struct {
PyObject_HEAD
@@ -6,6 +8,8 @@
Py_ssize_t pos;
Py_ssize_t string_size;
size_t buf_size;
+ PyObject *dict;
+ PyObject *weakreflist;
} BytesIOObject;
#define CHECK_CLOSED(self) \
@@ -144,10 +148,12 @@
static PyObject *
bytesio_get_closed(BytesIOObject *self)
{
- if (self->buf == NULL)
+ if (self->buf == NULL) {
Py_RETURN_TRUE;
- else
+ }
+ else {
Py_RETURN_FALSE;
+ }
}
/* Generic getter for the writable, readable and seekable properties */
@@ -532,22 +538,22 @@
static PyObject *
bytesio_write(BytesIOObject *self, PyObject *obj)
{
- const char *bytes;
- Py_ssize_t size;
Py_ssize_t n = 0;
+ Py_buffer buf;
+ PyObject *result = NULL;
CHECK_CLOSED(self);
- if (PyObject_AsReadBuffer(obj, (void *)&bytes, &size) < 0)
+ if (PyObject_GetBuffer(obj, &buf, PyBUF_CONTIG_RO) < 0)
return NULL;
- if (size != 0) {
- n = write_bytes(self, bytes, size);
- if (n < 0)
- return NULL;
- }
+ if (buf.len != 0)
+ n = write_bytes(self, buf.buf, buf.len);
+ if (n >= 0)
+ result = PyLong_FromSsize_t(n);
- return PyLong_FromSsize_t(n);
+ PyBuffer_Release(&buf);
+ return result;
}
PyDoc_STRVAR(writelines_doc,
@@ -607,6 +613,7 @@
PyMem_Free(self->buf);
self->buf = NULL;
}
+ Py_TYPE(self)->tp_clear((PyObject *)self);
Py_TYPE(self)->tp_free(self);
}
@@ -656,6 +663,24 @@
return 0;
}
+static int
+bytesio_traverse(BytesIOObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->dict);
+ Py_VISIT(self->weakreflist);
+ return 0;
+}
+
+static int
+bytesio_clear(BytesIOObject *self)
+{
+ Py_CLEAR(self->dict);
+ if (self->weakreflist != NULL)
+ PyObject_ClearWeakRefs((PyObject *)self);
+ return 0;
+}
+
+
static PyGetSetDef bytesio_getsetlist[] = {
{"closed", (getter)bytesio_get_closed, NULL,
"True if the file is closed."},
@@ -689,9 +714,9 @@
"Create a buffered I/O implementation using an in-memory bytes\n"
"buffer, ready for reading and writing.");
-static PyTypeObject BytesIO_Type = {
+PyTypeObject PyBytesIO_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "_bytesio._BytesIO", /*tp_name*/
+ "_io.BytesIO", /*tp_name*/
sizeof(BytesIOObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)bytesio_dealloc, /*tp_dealloc*/
@@ -709,12 +734,13 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
+ Py_TPFLAGS_HAVE_GC, /*tp_flags*/
bytesio_doc, /*tp_doc*/
- 0, /*tp_traverse*/
- 0, /*tp_clear*/
+ (traverseproc)bytesio_traverse, /*tp_traverse*/
+ (inquiry)bytesio_clear, /*tp_clear*/
0, /*tp_richcompare*/
- 0, /*tp_weaklistoffset*/
+ offsetof(BytesIOObject, weakreflist), /*tp_weaklistoffset*/
PyObject_SelfIter, /*tp_iter*/
(iternextfunc)bytesio_iternext, /*tp_iternext*/
bytesio_methods, /*tp_methods*/
@@ -724,36 +750,8 @@
0, /*tp_dict*/
0, /*tp_descr_get*/
0, /*tp_descr_set*/
- 0, /*tp_dictoffset*/
+ offsetof(BytesIOObject, dict), /*tp_dictoffset*/
(initproc)bytesio_init, /*tp_init*/
0, /*tp_alloc*/
bytesio_new, /*tp_new*/
};
-
-
-static struct PyModuleDef _bytesiomodule = {
- PyModuleDef_HEAD_INIT,
- "_bytesio",
- NULL,
- -1,
- NULL,
- NULL,
- NULL,
- NULL,
- NULL
-};
-
-PyMODINIT_FUNC
-PyInit__bytesio(void)
-{
- PyObject *m;
-
- if (PyType_Ready(&BytesIO_Type) < 0)
- return NULL;
- m = PyModule_Create(&_bytesiomodule);
- if (m == NULL)
- return NULL;
- Py_INCREF(&BytesIO_Type);
- PyModule_AddObject(m, "_BytesIO", (PyObject *)&BytesIO_Type);
- return m;
-}
Modified: python/branches/py3k/Modules/_fileio.c
==============================================================================
--- python/branches/py3k/Modules/_fileio.c (original)
+++ python/branches/py3k/Modules/_fileio.c Wed Mar 4 01:14:51 2009
@@ -6,6 +6,7 @@
#include <sys/stat.h>
#include <fcntl.h>
#include <stddef.h> /* For offsetof */
+#include "_iomodule.h"
/*
* Known likely problems:
@@ -49,14 +50,19 @@
int seekable : 2; /* -1 means unknown */
int closefd : 1;
PyObject *weakreflist;
+ PyObject *dict;
} PyFileIOObject;
PyTypeObject PyFileIO_Type;
#define PyFileIO_Check(op) (PyObject_TypeCheck((op), &PyFileIO_Type))
-static PyObject *
-portable_lseek(int fd, PyObject *posobj, int whence);
+
+int
+_PyFileIO_closed(PyObject *self)
+{
+ return ((PyFileIOObject *)self)->fd < 0;
+}
/* Returns 0 on success, -1 with exception set on failure. */
static int
@@ -88,14 +94,18 @@
self->fd = -1;
Py_RETURN_NONE;
}
- if (internal_close(self))
+ errno = internal_close(self);
+ if (errno < 0) {
+ PyErr_SetFromErrno(PyExc_IOError);
return NULL;
+ }
- Py_RETURN_NONE;
+ return PyObject_CallMethod((PyObject*)&PyRawIOBase_Type,
+ "close", "O", self);
}
static PyObject *
-fileio_new(PyTypeObject *type, PyObject *args, PyObject *kews)
+fileio_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
PyFileIOObject *self;
@@ -119,7 +129,7 @@
directories, so we need a check. */
static int
-dircheck(PyFileIOObject* self, char *name)
+dircheck(PyFileIOObject* self, const char *name)
{
#if defined(HAVE_FSTAT) && defined(S_IFDIR) && defined(EISDIR)
struct stat buf;
@@ -146,7 +156,7 @@
{
#if defined(HAVE_FSTAT)
struct stat buf;
- if (!_PyVerify_fd(fd) || (fstat(fd, &buf) < 0 && errno == EBADF)) {
+ if (fstat(fd, &buf) < 0 && errno == EBADF) {
PyObject *exc;
char *msg = strerror(EBADF);
exc = PyObject_CallFunction(PyExc_OSError, "(is)",
@@ -165,7 +175,8 @@
{
PyFileIOObject *self = (PyFileIOObject *) oself;
static char *kwlist[] = {"file", "mode", "closefd", NULL};
- char *name = NULL;
+ const char *name = NULL;
+ PyObject *nameobj;
char *mode = "r";
char *s;
#ifdef MS_WINDOWS
@@ -184,42 +195,58 @@
return -1;
}
- if (PyArg_ParseTupleAndKeywords(args, kwds, "i|si:fileio",
- kwlist, &fd, &mode, &closefd)) {
- if (fd < 0) {
+ if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|si:fileio",
+ kwlist, &nameobj, &mode, &closefd))
+ return -1;
+
+ if (PyFloat_Check(nameobj)) {
+ PyErr_SetString(PyExc_TypeError,
+ "integer argument expected, got float");
+ return -1;
+ }
+
+ fd = PyLong_AsLong(nameobj);
+ if (fd < 0) {
+ if (!PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Negative filedescriptor");
return -1;
}
- if (check_fd(fd))
- return -1;
- }
- else {
PyErr_Clear();
+ }
#ifdef Py_WIN_WIDE_FILENAMES
- if (GetVersion() < 0x80000000) {
+ if (GetVersion() < 0x80000000) {
/* On NT, so wide API available */
- PyObject *po;
- if (PyArg_ParseTupleAndKeywords(args, kwds, "U|si:fileio",
- kwlist, &po, &mode, &closefd)
- ) {
- widename = PyUnicode_AS_UNICODE(po);
- } else {
- /* Drop the argument parsing error as narrow
- strings are also valid. */
- PyErr_Clear();
- }
- }
- if (widename == NULL)
-#endif
- {
- if (!PyArg_ParseTupleAndKeywords(args, kwds, "et|si:fileio",
- kwlist,
- Py_FileSystemDefaultEncoding,
- &name, &mode, &closefd))
- return -1;
- }
+ if (PyUnicode_Check(nameobj))
+ widename = PyUnicode_AS_UNICODE(nameobj);
+ }
+ if (widename == NULL)
+#endif
+ if (fd < 0)
+ {
+ if (PyBytes_Check(nameobj) || PyByteArray_Check(nameobj)) {
+ if (PyObject_AsCharBuffer(nameobj, &name, NULL) < 0)
+ return -1;
+ }
+ else {
+ PyObject *s;
+ PyObject *u = PyUnicode_FromObject(nameobj);
+
+ if (u == NULL)
+ return -1;
+
+ s = PyUnicode_AsEncodedString(
+ u, Py_FileSystemDefaultEncoding, NULL);
+ Py_DECREF(u);
+ if (s == NULL)
+ return -1;
+ if (!PyBytes_Check(s)) {
+ PyErr_SetString(PyExc_TypeError,
+ "encoder failed to return bytes");
+ }
+ name = PyBytes_AS_STRING(s);
+ }
}
s = mode;
@@ -287,6 +314,8 @@
if (fd >= 0) {
self->fd = fd;
self->closefd = closefd;
+ if (check_fd(fd))
+ goto error;
}
else {
self->closefd = 1;
@@ -318,15 +347,8 @@
goto error;
}
- if (append) {
- /* For consistent behaviour, we explicitly seek to the
- end of file (otherwise, it might be done only on the
- first write()). */
- PyObject *pos = portable_lseek(self->fd, NULL, 2);
- if (pos == NULL)
- goto error;
- Py_DECREF(pos);
- }
+ if (PyObject_SetAttrString((PyObject *)self, "name", nameobj) < 0)
+ goto error;
goto done;
@@ -334,21 +356,32 @@
ret = -1;
done:
- PyMem_Free(name);
return ret;
}
+static int
+fileio_traverse(PyFileIOObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+fileio_clear(PyFileIOObject *self)
+{
+ Py_CLEAR(self->dict);
+ return 0;
+}
+
static void
fileio_dealloc(PyFileIOObject *self)
{
+ if (_PyIOBase_finalize((PyObject *) self) < 0)
+ return;
+ _PyObject_GC_UNTRACK(self);
if (self->weakreflist != NULL)
PyObject_ClearWeakRefs((PyObject *) self);
-
- if (self->fd >= 0 && self->closefd) {
- if(internal_close(self))
- PyErr_WriteUnraisable((PyObject*)self);
- }
-
+ Py_CLEAR(self->dict);
Py_TYPE(self)->tp_free((PyObject *)self);
}
@@ -437,6 +470,31 @@
return PyLong_FromSsize_t(n);
}
+static size_t
+new_buffersize(PyFileIOObject *self, size_t currentsize)
+{
+#ifdef HAVE_FSTAT
+ off_t pos, end;
+ struct stat st;
+ if (fstat(self->fd, &st) == 0) {
+ end = st.st_size;
+ pos = lseek(self->fd, 0L, SEEK_CUR);
+ if (end >= pos && pos >= 0)
+ return currentsize + end - pos + 1;
+ /* Add 1 so if the file were to grow we'd notice. */
+ }
+#endif
+ if (currentsize > SMALLCHUNK) {
+ /* Keep doubling until we reach BIGCHUNK;
+ then keep adding BIGCHUNK. */
+ if (currentsize <= BIGCHUNK)
+ return currentsize + currentsize;
+ else
+ return currentsize + BIGCHUNK;
+ }
+ return currentsize + SMALLCHUNK;
+}
+
static PyObject *
fileio_readall(PyFileIOObject *self)
{
@@ -449,17 +507,7 @@
return NULL;
while (1) {
- Py_ssize_t newsize = (total < SMALLCHUNK) ? SMALLCHUNK : total;
-
- /* Keep doubling until we reach BIGCHUNK;
- then keep adding BIGCHUNK. */
- if (newsize <= BIGCHUNK) {
- newsize += newsize;
- }
- else {
- /* NOTE: overflow impossible due to limits on BUFSIZ */
- newsize += BIGCHUNK;
- }
+ Py_ssize_t newsize = new_buffersize(self, total);
if (PyBytes_GET_SIZE(result) < newsize) {
if (_PyBytes_Resize(&result, newsize) < 0) {
@@ -583,12 +631,6 @@
/* XXX Windows support below is likely incomplete */
-#if defined(MS_WIN64) || defined(MS_WINDOWS)
-typedef PY_LONG_LONG Py_off_t;
-#else
-typedef off_t Py_off_t;
-#endif
-
/* Cribbed from posix_lseek() */
static PyObject *
portable_lseek(int fd, PyObject *posobj, int whence)
@@ -755,9 +797,9 @@
fileio_repr(PyFileIOObject *self)
{
if (self->fd < 0)
- return PyUnicode_FromFormat("_fileio._FileIO(-1)");
+ return PyUnicode_FromFormat("io.FileIO(-1)");
- return PyUnicode_FromFormat("_fileio._FileIO(%d, '%s')",
+ return PyUnicode_FromFormat("io.FileIO(%d, '%s')",
self->fd, mode_string(self));
}
@@ -899,15 +941,15 @@
};
PyTypeObject PyFileIO_Type = {
- PyVarObject_HEAD_INIT(&PyType_Type, 0)
- "_FileIO",
+ PyVarObject_HEAD_INIT(NULL, 0)
+ "FileIO",
sizeof(PyFileIOObject),
0,
(destructor)fileio_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
- 0, /* tp_reserved */
+ 0, /* tp_compare */
(reprfunc)fileio_repr, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
@@ -918,10 +960,11 @@
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /* tp_flags */
fileio_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)fileio_traverse, /* tp_traverse */
+ (inquiry)fileio_clear, /* tp_clear */
0, /* tp_richcompare */
offsetof(PyFileIOObject, weakreflist), /* tp_weaklistoffset */
0, /* tp_iter */
@@ -933,40 +976,9 @@
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
- 0, /* tp_dictoffset */
+ offsetof(PyFileIOObject, dict), /* tp_dictoffset */
fileio_init, /* tp_init */
PyType_GenericAlloc, /* tp_alloc */
fileio_new, /* tp_new */
- PyObject_Del, /* tp_free */
-};
-
-static PyMethodDef module_methods[] = {
- {NULL, NULL}
-};
-
-static struct PyModuleDef fileiomodule = {
- PyModuleDef_HEAD_INIT,
- "_fileio",
- "Fast implementation of io.FileIO.",
- -1,
- module_methods,
- NULL,
- NULL,
- NULL,
- NULL
+ PyObject_GC_Del, /* tp_free */
};
-
-PyMODINIT_FUNC
-PyInit__fileio(void)
-{
- PyObject *m; /* a module object */
-
- m = PyModule_Create(&fileiomodule);
- if (m == NULL)
- return NULL;
- if (PyType_Ready(&PyFileIO_Type) < 0)
- return NULL;
- Py_INCREF(&PyFileIO_Type);
- PyModule_AddObject(m, "_FileIO", (PyObject *) &PyFileIO_Type);
- return m;
-}
Copied: python/branches/py3k/Modules/_iobase.c (from r68756, /python/branches/io-c/Modules/_iobase.c)
==============================================================================
--- /python/branches/io-c/Modules/_iobase.c (original)
+++ python/branches/py3k/Modules/_iobase.c Wed Mar 4 01:14:51 2009
@@ -1,3 +1,13 @@
+/*
+ An implementation of the I/O abstract base classes hierarchy
+ as defined by PEP 3116 - "New I/O"
+
+ Classes defined here: IOBase, RawIOBase.
+
+ Written by Amaury Forgeot d'Arc and Antoine Pitrou
+*/
+
+
#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
@@ -7,6 +17,13 @@
* IOBase class, an abstract class
*/
+typedef struct {
+ PyObject_HEAD
+
+ PyObject *dict;
+ PyObject *weakreflist;
+} IOBaseObject;
+
PyDoc_STRVAR(IOBase_doc,
"The abstract base class for all I/O classes, acting on streams of\n"
"bytes. There is no public constructor.\n"
@@ -48,7 +65,7 @@
static PyObject *
IOBase_unsupported(const char *message)
{
- PyErr_SetString(PyIOExc_UnsupportedOperation, message);
+ PyErr_SetString(IO_STATE->unsupported_operation, message);
return NULL;
}
@@ -91,7 +108,7 @@
static PyObject *
IOBase_truncate(PyObject *self, PyObject *args)
{
- return IOBase_unsupported("seek");
+ return IOBase_unsupported("truncate");
}
/* Flush and close methods */
@@ -176,7 +193,7 @@
Py_RETURN_NONE;
}
-/* Destructor */
+/* Finalization and garbage collection support */
int
_PyIOBase_finalize(PyObject *self)
@@ -184,12 +201,17 @@
PyObject *res;
PyObject *tp, *v, *tb;
int closed = 1;
+ int is_zombie;
+
+ /* If _PyIOBase_finalize() is called from a destructor, we need to
+ resurrect the object as calling close() can invoke arbitrary code. */
+ is_zombie = (Py_REFCNT(self) == 0);
+ if (is_zombie) {
+ ++Py_REFCNT(self);
+ }
PyErr_Fetch(&tp, &v, &tb);
- /* We need to resurrect the object as calling close() can invoke
- arbitrary code. */
- ((PyObject *) self)->ob_refcnt++;
- /* The object could already be in an usable state, so we'll take any
- error as meaning "stop, nothing to see here". */
+ /* If `closed` doesn't exist or can't be evaluated as bool, then the
+ object is probably in an unusable state, so ignore. */
res = PyObject_GetAttr(self, _PyIO_str_closed);
if (res == NULL)
PyErr_Clear();
@@ -202,22 +224,79 @@
if (closed == 0) {
res = PyObject_CallMethodObjArgs((PyObject *) self, _PyIO_str_close,
NULL);
+ /* Silencing I/O errors is bad, but printing spurious tracebacks is
+ equally as bad, and potentially more frequent (because of
+ shutdown issues). */
if (res == NULL)
PyErr_Clear();
else
Py_DECREF(res);
}
PyErr_Restore(tp, v, tb);
- if (--((PyObject *) self)->ob_refcnt != 0)
+ if (is_zombie) {
+ if (--Py_REFCNT(self) != 0) {
+ /* The object lives again. The following code is taken from
+ slot_tp_del in typeobject.c. */
+ Py_ssize_t refcnt = Py_REFCNT(self);
+ _Py_NewReference(self);
+ Py_REFCNT(self) = refcnt;
+ /* If Py_REF_DEBUG, _Py_NewReference bumped _Py_RefTotal, so
+ * we need to undo that. */
+ _Py_DEC_REFTOTAL;
+ /* If Py_TRACE_REFS, _Py_NewReference re-added self to the object
+ * chain, so no more to do there.
+ * If COUNT_ALLOCS, the original decref bumped tp_frees, and
+ * _Py_NewReference bumped tp_allocs: both of those need to be
+ * undone.
+ */
+#ifdef COUNT_ALLOCS
+ --Py_TYPE(self)->tp_frees;
+ --Py_TYPE(self)->tp_allocs;
+#endif
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int
+IOBase_traverse(IOBaseObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+IOBase_clear(IOBaseObject *self)
+{
+ if (_PyIOBase_finalize((PyObject *) self) < 0)
return -1;
+ Py_CLEAR(self->dict);
return 0;
}
+/* Destructor */
+
static void
-IOBase_dealloc(PyObject *self)
+IOBase_dealloc(IOBaseObject *self)
{
- if (_PyIOBase_finalize(self) == 0)
- Py_TYPE(self)->tp_free(self);
+ /* NOTE: since IOBaseObject has its own dict, Python-defined attributes
+ are still available here for close() to use.
+ However, if the derived class declares a __slots__, those slots are
+ already gone.
+ */
+ if (_PyIOBase_finalize((PyObject *) self) < 0) {
+ /* When called from a heap type's dealloc, the type will be
+ decref'ed on return (see e.g. subtype_dealloc in typeobject.c). */
+ if (PyType_HasFeature(Py_TYPE(self), Py_TPFLAGS_HEAPTYPE))
+ Py_INCREF(Py_TYPE(self));
+ return;
+ }
+ _PyObject_GC_UNTRACK(self);
+ if (self->weakreflist != NULL)
+ PyObject_ClearWeakRefs((PyObject *) self);
+ Py_CLEAR(self->dict);
+ Py_TYPE(self)->tp_free((PyObject *) self);
}
/* Inquiry methods */
@@ -517,8 +596,11 @@
return NULL;
if (hint <= 0) {
+ /* XXX special-casing this made sense in the Python version in order
+ to remove the bytecode interpretation overhead, but it could
+ probably be removed here. */
PyObject *ret = PyObject_CallMethod(result, "extend", "O", self);
- if( ret == NULL) {
+ if (ret == NULL) {
Py_DECREF(result);
return NULL;
}
@@ -534,7 +616,7 @@
return NULL;
}
else
- break; /* SopIteration raised */
+ break; /* StopIteration raised */
}
if (PyList_Append(result, line) < 0) {
@@ -569,7 +651,7 @@
while (1) {
PyObject *line = PyIter_Next(iter);
- if(line == NULL) {
+ if (line == NULL) {
if (PyErr_Occurred()) {
Py_DECREF(iter);
return NULL;
@@ -586,6 +668,7 @@
}
Py_DECREF(res);
}
+ Py_DECREF(iter);
Py_RETURN_NONE;
}
@@ -626,10 +709,10 @@
PyTypeObject PyIOBase_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "IOBase", /*tp_name*/
- 0, /*tp_basicsize*/
+ "_io._IOBase", /*tp_name*/
+ sizeof(IOBaseObject), /*tp_basicsize*/
0, /*tp_itemsize*/
- IOBase_dealloc, /*tp_dealloc*/
+ (destructor)IOBase_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
@@ -644,12 +727,13 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
IOBase_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)IOBase_traverse, /* tp_traverse */
+ (inquiry)IOBase_clear, /* tp_clear */
0, /* tp_richcompare */
- 0, /* tp_weaklistoffset */
+ offsetof(IOBaseObject, weakreflist), /* tp_weaklistoffset */
IOBase_iter, /* tp_iter */
IOBase_iternext, /* tp_iternext */
IOBase_methods, /* tp_methods */
@@ -659,21 +743,12 @@
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
- 0, /* tp_dictoffset */
+ offsetof(IOBaseObject, dict), /* tp_dictoffset */
0, /* tp_init */
0, /* tp_alloc */
PyType_GenericNew, /* tp_new */
- 0, /* tp_free */
- 0, /* tp_is_gc */
- 0, /* tp_bases */
- 0, /* tp_mro */
- 0, /* tp_cache */
- 0, /* tp_subclasses */
- 0, /* tp_weaklist */
- 0, /* tp_del */
};
-
/*
* RawIOBase class, Inherits from IOBase.
@@ -790,7 +865,7 @@
PyTypeObject PyRawIOBase_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "RawIOBase", /*tp_name*/
+ "_io._RawIOBase", /*tp_name*/
0, /*tp_basicsize*/
0, /*tp_itemsize*/
0, /*tp_dealloc*/
Copied: python/branches/py3k/Modules/_iomodule.h (from r68756, /python/branches/io-c/Modules/_iomodule.h)
==============================================================================
--- /python/branches/io-c/Modules/_iomodule.h (original)
+++ python/branches/py3k/Modules/_iomodule.h Wed Mar 4 01:14:51 2009
@@ -2,11 +2,16 @@
* Declarations shared between the different parts of the io module
*/
-extern PyTypeObject PyFileIO_Type;
-extern PyTypeObject PyBytesIO_Type;
+/* ABCs */
extern PyTypeObject PyIOBase_Type;
extern PyTypeObject PyRawIOBase_Type;
extern PyTypeObject PyBufferedIOBase_Type;
+extern PyTypeObject PyTextIOBase_Type;
+
+/* Concrete classes */
+extern PyTypeObject PyFileIO_Type;
+extern PyTypeObject PyBytesIO_Type;
+extern PyTypeObject PyStringIO_Type;
extern PyTypeObject PyBufferedReader_Type;
extern PyTypeObject PyBufferedWriter_Type;
extern PyTypeObject PyBufferedRWPair_Type;
@@ -23,14 +28,33 @@
extern PyObject* _PyIOBase_checkSeekable(PyObject *self, PyObject *args);
extern PyObject* _PyIOBase_checkClosed(PyObject *self, PyObject *args);
-extern PyObject* PyIOExc_UnsupportedOperation;
-
/* Helper for finalization.
This function will revive an object ready to be deallocated and try to
close() it. It returns 0 if the object can be destroyed, or -1 if it
is alive again. */
extern int _PyIOBase_finalize(PyObject *self);
+/* Returns true if the given FileIO object is closed.
+ Doesn't check the argument type, so be careful! */
+extern int _PyFileIO_closed(PyObject *self);
+
+/* Shortcut to the core of the IncrementalNewlineDecoder.decode method */
+extern PyObject *_PyIncrementalNewlineDecoder_decode(
+ PyObject *self, PyObject *input, int final);
+
+/* Finds the first line ending between `start` and `end`.
+ If found, returns the index after the line ending and doesn't touch
+ `*consumed`.
+ If not found, returns -1 and sets `*consumed` to the number of characters
+ which can be safely put aside until another search.
+
+ NOTE: for performance reasons, `end` must point to a NUL character ('\0').
+ Otherwise, the function will scan further and return garbage. */
+extern Py_ssize_t _PyIO_find_line_ending(
+ int translated, int universal, PyObject *readnl,
+ Py_UNICODE *start, Py_UNICODE *end, Py_ssize_t *consumed);
+
+
#define DEFAULT_BUFFER_SIZE (8 * 1024) /* bytes */
typedef struct {
@@ -84,6 +108,21 @@
/* Implementation details */
+/* IO module structure */
+
+extern PyModuleDef _PyIO_Module;
+
+typedef struct {
+ int initialized;
+ PyObject *os_module;
+ PyObject *locale_module;
+
+ PyObject *unsupported_operation;
+} _PyIO_State;
+
+#define IO_MOD_STATE(mod) ((_PyIO_State *)PyModule_GetState(mod))
+#define IO_STATE IO_MOD_STATE(PyState_FindModule(&_PyIO_Module))
+
extern PyObject *_PyIO_str_close;
extern PyObject *_PyIO_str_closed;
extern PyObject *_PyIO_str_decode;
@@ -93,14 +132,19 @@
extern PyObject *_PyIO_str_getstate;
extern PyObject *_PyIO_str_isatty;
extern PyObject *_PyIO_str_newlines;
+extern PyObject *_PyIO_str_nl;
extern PyObject *_PyIO_str_read;
extern PyObject *_PyIO_str_read1;
extern PyObject *_PyIO_str_readable;
extern PyObject *_PyIO_str_readinto;
extern PyObject *_PyIO_str_readline;
+extern PyObject *_PyIO_str_reset;
extern PyObject *_PyIO_str_seek;
extern PyObject *_PyIO_str_seekable;
extern PyObject *_PyIO_str_tell;
extern PyObject *_PyIO_str_truncate;
extern PyObject *_PyIO_str_writable;
extern PyObject *_PyIO_str_write;
+
+extern PyObject *_PyIO_empty_str;
+extern PyObject *_PyIO_empty_bytes;
Modified: python/branches/py3k/Modules/_stringio.c
==============================================================================
--- python/branches/py3k/Modules/_stringio.c (original)
+++ python/branches/py3k/Modules/_stringio.c Wed Mar 4 01:14:51 2009
@@ -1,8 +1,11 @@
+#define PY_SSIZE_T_CLEAN
#include "Python.h"
+#include "structmember.h"
+#include "_iomodule.h"
-/* This module is a stripped down version of _bytesio.c with a Py_UNICODE
- buffer. Most of the functionality is provided by subclassing _StringIO. */
-
+/* Implementation note: the buffer is always at least one character longer
+ than the enclosed string, for proper functioning of _PyIO_find_line_ending.
+*/
typedef struct {
PyObject_HEAD
@@ -10,8 +13,39 @@
Py_ssize_t pos;
Py_ssize_t string_size;
size_t buf_size;
+
+ char ok; /* initialized? */
+ char closed;
+ char readuniversal;
+ char readtranslate;
+ PyObject *decoder;
+ PyObject *readnl;
+ PyObject *writenl;
+
+ PyObject *dict;
+ PyObject *weakreflist;
} StringIOObject;
+#define CHECK_INITIALIZED(self) \
+ if (self->ok <= 0) { \
+ PyErr_SetString(PyExc_ValueError, \
+ "I/O operation on uninitialized object"); \
+ return NULL; \
+ }
+
+#define CHECK_CLOSED(self) \
+ if (self->closed) { \
+ PyErr_SetString(PyExc_ValueError, \
+ "I/O operation on closed file"); \
+ return NULL; \
+ }
+
+PyDoc_STRVAR(stringio_doc,
+ "Text I/O implementation using an in-memory buffer.\n"
+ "\n"
+ "The initial_value argument sets the value of object. The newline\n"
+ "argument is like the one of TextIOWrapper's constructor.");
+
/* Internal routine for changing the size, in terms of characters, of the
buffer of StringIO objects. The caller should ensure that the 'size'
@@ -26,6 +60,8 @@
assert(self->buf != NULL);
+ /* Reserve one more char for line ending detection. */
+ size = size + 1;
/* For simplicity, stay in the range of the signed type. Anyway, Python
doesn't allow strings to be longer than this. */
if (size > PY_SSIZE_T_MAX)
@@ -67,13 +103,38 @@
return -1;
}
-/* Internal routine for writing a string of characters to the buffer of a
- StringIO object. Returns the number of bytes wrote, or -1 on error. */
+/* Internal routine for writing a whole PyUnicode object to the buffer of a
+ StringIO object. Returns 0 on success, or -1 on error. */
static Py_ssize_t
-write_str(StringIOObject *self, const Py_UNICODE *str, Py_ssize_t len)
+write_str(StringIOObject *self, PyObject *obj)
{
+ Py_UNICODE *str;
+ Py_ssize_t len;
+ PyObject *decoded = NULL;
assert(self->buf != NULL);
assert(self->pos >= 0);
+
+ if (self->decoder != NULL) {
+ decoded = _PyIncrementalNewlineDecoder_decode(
+ self->decoder, obj, 1 /* always final */);
+ }
+ else {
+ decoded = obj;
+ Py_INCREF(decoded);
+ }
+ if (self->writenl) {
+ PyObject *translated = PyUnicode_Replace(
+ decoded, _PyIO_str_nl, self->writenl, -1);
+ Py_DECREF(decoded);
+ decoded = translated;
+ }
+ if (decoded == NULL)
+ return -1;
+
+ assert(PyUnicode_Check(decoded));
+ str = PyUnicode_AS_UNICODE(decoded);
+ len = PyUnicode_GET_SIZE(decoded);
+
assert(len >= 0);
/* This overflow check is not strictly necessary. However, it avoids us to
@@ -82,11 +143,11 @@
if (self->pos > PY_SSIZE_T_MAX - len) {
PyErr_SetString(PyExc_OverflowError,
"new position too large");
- return -1;
+ goto fail;
}
if (self->pos + len > self->string_size) {
if (resize_buffer(self, self->pos + len) < 0)
- return -1;
+ goto fail;
}
if (self->pos > self->string_size) {
@@ -108,26 +169,47 @@
memcpy(self->buf + self->pos, str, len * sizeof(Py_UNICODE));
self->pos += len;
- /* Set the new length of the internal string if it has changed */
+ /* Set the new length of the internal string if it has changed. */
if (self->string_size < self->pos) {
self->string_size = self->pos;
}
- return len;
+ Py_DECREF(decoded);
+ return 0;
+
+fail:
+ Py_XDECREF(decoded);
+ return -1;
}
+PyDoc_STRVAR(stringio_getvalue_doc,
+ "Retrieve the entire contents of the object.");
+
static PyObject *
stringio_getvalue(StringIOObject *self)
{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
return PyUnicode_FromUnicode(self->buf, self->string_size);
}
+PyDoc_STRVAR(stringio_tell_doc,
+ "Tell the current file position.");
+
static PyObject *
stringio_tell(StringIOObject *self)
{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
return PyLong_FromSsize_t(self->pos);
}
+PyDoc_STRVAR(stringio_read_doc,
+ "Read at most n characters, returned as a string.\n"
+ "\n"
+ "If the argument is negative or omitted, read until EOF\n"
+ "is reached. Return an empty string at EOF.\n");
+
static PyObject *
stringio_read(StringIOObject *self, PyObject *args)
{
@@ -135,11 +217,13 @@
Py_UNICODE *output;
PyObject *arg = Py_None;
+ CHECK_INITIALIZED(self);
if (!PyArg_ParseTuple(args, "|O:read", &arg))
return NULL;
+ CHECK_CLOSED(self);
- if (PyLong_Check(arg)) {
- size = PyLong_AsSsize_t(arg);
+ if (PyNumber_Check(arg)) {
+ size = PyNumber_AsSsize_t(arg, PyExc_OverflowError);
if (size == -1 && PyErr_Occurred())
return NULL;
}
@@ -161,24 +245,127 @@
size = 0;
}
- assert(self->buf != NULL);
output = self->buf + self->pos;
self->pos += size;
-
return PyUnicode_FromUnicode(output, size);
}
+/* Internal helper, used by stringio_readline and stringio_iternext */
+static PyObject *
+_stringio_readline(StringIOObject *self, Py_ssize_t limit)
+{
+ Py_UNICODE *start, *end, old_char;
+ Py_ssize_t len, consumed;
+
+ /* In case of overseek, return the empty string */
+ if (self->pos >= self->string_size)
+ return PyUnicode_FromString("");
+
+ start = self->buf + self->pos;
+ if (limit < 0 || limit > self->string_size - self->pos)
+ limit = self->string_size - self->pos;
+
+ end = start + limit;
+ old_char = *end;
+ *end = '\0';
+ len = _PyIO_find_line_ending(
+ self->readtranslate, self->readuniversal, self->readnl,
+ start, end, &consumed);
+ *end = old_char;
+ /* If we haven't found any line ending, we just return everything
+ (`consumed` is ignored). */
+ if (len < 0)
+ len = limit;
+ self->pos += len;
+ return PyUnicode_FromUnicode(start, len);
+}
+
+PyDoc_STRVAR(stringio_readline_doc,
+ "Read until newline or EOF.\n"
+ "\n"
+ "Returns an empty string if EOF is hit immediately.\n");
+
+static PyObject *
+stringio_readline(StringIOObject *self, PyObject *args)
+{
+ PyObject *arg = Py_None;
+ Py_ssize_t limit = -1;
+
+ CHECK_INITIALIZED(self);
+ if (!PyArg_ParseTuple(args, "|O:readline", &arg))
+ return NULL;
+ CHECK_CLOSED(self);
+
+ if (PyNumber_Check(arg)) {
+ limit = PyNumber_AsSsize_t(arg, PyExc_OverflowError);
+ if (limit == -1 && PyErr_Occurred())
+ return NULL;
+ }
+ else if (arg != Py_None) {
+ PyErr_Format(PyExc_TypeError, "integer argument expected, got '%s'",
+ Py_TYPE(arg)->tp_name);
+ return NULL;
+ }
+ return _stringio_readline(self, limit);
+}
+
+static PyObject *
+stringio_iternext(StringIOObject *self)
+{
+ PyObject *line;
+
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
+
+ if (Py_TYPE(self) == &PyStringIO_Type) {
+ /* Skip method call overhead for speed */
+ line = _stringio_readline(self, -1);
+ }
+ else {
+ /* XXX is subclassing StringIO really supported? */
+ line = PyObject_CallMethodObjArgs((PyObject *)self,
+ _PyIO_str_readline, NULL);
+ if (line && !PyUnicode_Check(line)) {
+ PyErr_Format(PyExc_IOError,
+ "readline() should have returned an str object, "
+ "not '%.200s'", Py_TYPE(line)->tp_name);
+ Py_DECREF(line);
+ return NULL;
+ }
+ }
+
+ if (line == NULL)
+ return NULL;
+
+ if (PyUnicode_GET_SIZE(line) == 0) {
+ /* Reached EOF */
+ Py_DECREF(line);
+ return NULL;
+ }
+
+ return line;
+}
+
+PyDoc_STRVAR(stringio_truncate_doc,
+ "Truncate size to pos.\n"
+ "\n"
+ "The pos argument defaults to the current file position, as\n"
+ "returned by tell(). Imply an absolute seek to pos.\n"
+ "Returns the new absolute position.\n");
+
static PyObject *
stringio_truncate(StringIOObject *self, PyObject *args)
{
Py_ssize_t size;
PyObject *arg = Py_None;
+ CHECK_INITIALIZED(self);
if (!PyArg_ParseTuple(args, "|O:truncate", &arg))
return NULL;
+ CHECK_CLOSED(self);
- if (PyLong_Check(arg)) {
- size = PyLong_AsSsize_t(arg);
+ if (PyNumber_Check(arg)) {
+ size = PyNumber_AsSsize_t(arg, PyExc_OverflowError);
if (size == -1 && PyErr_Occurred())
return NULL;
}
@@ -199,23 +386,34 @@
}
if (size < self->string_size) {
- self->string_size = size;
if (resize_buffer(self, size) < 0)
return NULL;
+ self->string_size = size;
}
self->pos = size;
return PyLong_FromSsize_t(size);
}
+PyDoc_STRVAR(stringio_seek_doc,
+ "Change stream position.\n"
+ "\n"
+ "Seek to character offset pos relative to position indicated by whence:\n"
+ " 0 Start of stream (the default). pos should be >= 0;\n"
+ " 1 Current position - pos must be 0;\n"
+ " 2 End of stream - pos must be 0.\n"
+ "Returns the new absolute position.\n");
+
static PyObject *
stringio_seek(StringIOObject *self, PyObject *args)
{
Py_ssize_t pos;
int mode = 0;
+ CHECK_INITIALIZED(self);
if (!PyArg_ParseTuple(args, "n|i:seek", &pos, &mode))
return NULL;
+ CHECK_CLOSED(self);
if (mode != 0 && mode != 1 && mode != 2) {
PyErr_Format(PyExc_ValueError,
@@ -248,36 +446,76 @@
return PyLong_FromSsize_t(self->pos);
}
+PyDoc_STRVAR(stringio_write_doc,
+ "Write string to file.\n"
+ "\n"
+ "Returns the number of characters written, which is always equal to\n"
+ "the length of the string.\n");
+
static PyObject *
stringio_write(StringIOObject *self, PyObject *obj)
{
- const Py_UNICODE *str;
Py_ssize_t size;
- Py_ssize_t n = 0;
- if (PyUnicode_Check(obj)) {
- str = PyUnicode_AsUnicode(obj);
- size = PyUnicode_GetSize(obj);
- }
- else {
+ CHECK_INITIALIZED(self);
+ if (!PyUnicode_Check(obj)) {
PyErr_Format(PyExc_TypeError, "string argument expected, got '%s'",
Py_TYPE(obj)->tp_name);
return NULL;
}
+ CHECK_CLOSED(self);
+ size = PyUnicode_GET_SIZE(obj);
- if (size != 0) {
- n = write_str(self, str, size);
- if (n < 0)
- return NULL;
- }
+ if (size > 0 && write_str(self, obj) < 0)
+ return NULL;
+
+ return PyLong_FromSsize_t(size);
+}
- return PyLong_FromSsize_t(n);
+PyDoc_STRVAR(stringio_close_doc,
+ "Close the IO object. Attempting any further operation after the\n"
+ "object is closed will raise a ValueError.\n"
+ "\n"
+ "This method has no effect if the file is already closed.\n");
+
+static PyObject *
+stringio_close(StringIOObject *self)
+{
+ self->closed = 1;
+ /* Free up some memory */
+ if (resize_buffer(self, 0) < 0)
+ return NULL;
+ Py_CLEAR(self->readnl);
+ Py_CLEAR(self->writenl);
+ Py_CLEAR(self->decoder);
+ Py_RETURN_NONE;
+}
+
+static int
+stringio_traverse(StringIOObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+stringio_clear(StringIOObject *self)
+{
+ Py_CLEAR(self->dict);
+ return 0;
}
static void
stringio_dealloc(StringIOObject *self)
{
- PyMem_Free(self->buf);
+ _PyObject_GC_UNTRACK(self);
+ Py_CLEAR(self->readnl);
+ Py_CLEAR(self->writenl);
+ Py_CLEAR(self->decoder);
+ if (self->buf)
+ PyMem_Free(self->buf);
+ if (self->weakreflist != NULL)
+ PyObject_ClearWeakRefs((PyObject *) self);
Py_TYPE(self)->tp_free(self);
}
@@ -303,19 +541,194 @@
return (PyObject *)self;
}
+static int
+stringio_init(StringIOObject *self, PyObject *args, PyObject *kwds)
+{
+ char *kwlist[] = {"initial_value", "newline", NULL};
+ PyObject *value = NULL;
+ char *newline = "\n";
+
+ if (!PyArg_ParseTupleAndKeywords(args, kwds, "|Oz:__init__", kwlist,
+ &value, &newline))
+ return -1;
+
+ if (newline && newline[0] != '\0'
+ && !(newline[0] == '\n' && newline[1] == '\0')
+ && !(newline[0] == '\r' && newline[1] == '\0')
+ && !(newline[0] == '\r' && newline[1] == '\n' && newline[2] == '\0')) {
+ PyErr_Format(PyExc_ValueError,
+ "illegal newline value: %s", newline);
+ return -1;
+ }
+ if (value && value != Py_None && !PyUnicode_Check(value)) {
+ PyErr_Format(PyExc_ValueError,
+ "initial_value must be str or None, not %.200s",
+ Py_TYPE(value)->tp_name);
+ return -1;
+ }
+
+ self->ok = 0;
+
+ Py_CLEAR(self->readnl);
+ Py_CLEAR(self->writenl);
+ Py_CLEAR(self->decoder);
+
+ if (newline) {
+ self->readnl = PyUnicode_FromString(newline);
+ if (self->readnl == NULL)
+ return -1;
+ }
+ self->readuniversal = (newline == NULL || newline[0] == '\0');
+ self->readtranslate = (newline == NULL);
+ /* If newline == "", we don't translate anything.
+ If newline == "\n" or newline == None, we translate to "\n", which is
+ a no-op.
+ (for newline == None, TextIOWrapper translates to os.sepline, but it
+ is pointless for StringIO)
+ */
+ if (newline != NULL && newline[0] == '\r') {
+ self->writenl = self->readnl;
+ Py_INCREF(self->writenl);
+ }
+
+ if (self->readuniversal) {
+ self->decoder = PyObject_CallFunction(
+ (PyObject *)&PyIncrementalNewlineDecoder_Type,
+ "Oi", Py_None, (int) self->readtranslate);
+ if (self->decoder == NULL)
+ return -1;
+ }
+
+ /* Now everything is set up, resize buffer to size of initial value,
+ and copy it */
+ self->string_size = 0;
+ if (value && value != Py_None) {
+ Py_ssize_t len = PyUnicode_GetSize(value);
+ /* This is a heuristic, for newline translation might change
+ the string length. */
+ if (resize_buffer(self, len) < 0)
+ return -1;
+ self->pos = 0;
+ if (write_str(self, value) < 0)
+ return -1;
+ }
+ else {
+ if (resize_buffer(self, 0) < 0)
+ return -1;
+ }
+ self->pos = 0;
+
+ self->closed = 0;
+ self->ok = 1;
+ return 0;
+}
+
+/* Properties and pseudo-properties */
+static PyObject *
+stringio_seekable(StringIOObject *self, PyObject *args)
+{
+ CHECK_INITIALIZED(self);
+ Py_RETURN_TRUE;
+}
+
+static PyObject *
+stringio_readable(StringIOObject *self, PyObject *args)
+{
+ CHECK_INITIALIZED(self);
+ Py_RETURN_TRUE;
+}
+
+static PyObject *
+stringio_writable(StringIOObject *self, PyObject *args)
+{
+ CHECK_INITIALIZED(self);
+ Py_RETURN_TRUE;
+}
+
+static PyObject *
+stringio_buffer(StringIOObject *self, void *context)
+{
+ PyErr_SetString(IO_STATE->unsupported_operation,
+ "buffer attribute is unsupported on type StringIO");
+ return NULL;
+}
+
+static PyObject *
+stringio_closed(StringIOObject *self, void *context)
+{
+ CHECK_INITIALIZED(self);
+ return PyBool_FromLong(self->closed);
+}
+
+static PyObject *
+stringio_encoding(StringIOObject *self, void *context)
+{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
+ return PyUnicode_FromString("utf-8");
+}
+
+static PyObject *
+stringio_errors(StringIOObject *self, void *context)
+{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
+ return PyUnicode_FromString("strict");
+}
+
+static PyObject *
+stringio_line_buffering(StringIOObject *self, void *context)
+{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
+ Py_RETURN_FALSE;
+}
+
+static PyObject *
+stringio_newlines(StringIOObject *self, void *context)
+{
+ CHECK_INITIALIZED(self);
+ CHECK_CLOSED(self);
+ if (self->decoder == NULL)
+ Py_RETURN_NONE;
+ return PyObject_GetAttr(self->decoder, _PyIO_str_newlines);
+}
+
static struct PyMethodDef stringio_methods[] = {
- {"getvalue", (PyCFunction)stringio_getvalue, METH_VARARGS, NULL},
- {"read", (PyCFunction)stringio_read, METH_VARARGS, NULL},
- {"tell", (PyCFunction)stringio_tell, METH_NOARGS, NULL},
- {"truncate", (PyCFunction)stringio_truncate, METH_VARARGS, NULL},
- {"seek", (PyCFunction)stringio_seek, METH_VARARGS, NULL},
- {"write", (PyCFunction)stringio_write, METH_O, NULL},
+ {"close", (PyCFunction)stringio_close, METH_NOARGS, stringio_close_doc},
+ {"getvalue", (PyCFunction)stringio_getvalue, METH_VARARGS, stringio_getvalue_doc},
+ {"read", (PyCFunction)stringio_read, METH_VARARGS, stringio_read_doc},
+ {"readline", (PyCFunction)stringio_readline, METH_VARARGS, stringio_readline_doc},
+ {"tell", (PyCFunction)stringio_tell, METH_NOARGS, stringio_tell_doc},
+ {"truncate", (PyCFunction)stringio_truncate, METH_VARARGS, stringio_truncate_doc},
+ {"seek", (PyCFunction)stringio_seek, METH_VARARGS, stringio_seek_doc},
+ {"write", (PyCFunction)stringio_write, METH_O, stringio_write_doc},
+
+ {"seekable", (PyCFunction)stringio_seekable, METH_NOARGS},
+ {"readable", (PyCFunction)stringio_readable, METH_NOARGS},
+ {"writable", (PyCFunction)stringio_writable, METH_NOARGS},
{NULL, NULL} /* sentinel */
};
-static PyTypeObject StringIO_Type = {
+static PyGetSetDef stringio_getset[] = {
+ {"closed", (getter)stringio_closed, NULL, NULL},
+ {"newlines", (getter)stringio_newlines, NULL, NULL},
+ /* (following comments straight off of the original Python wrapper:)
+ XXX Cruft to support the TextIOWrapper API. This would only
+ be meaningful if StringIO supported the buffer attribute.
+ Hopefully, a better solution, than adding these pseudo-attributes,
+ will be found.
+ */
+ {"buffer", (getter)stringio_buffer, NULL, NULL},
+ {"encoding", (getter)stringio_encoding, NULL, NULL},
+ {"errors", (getter)stringio_errors, NULL, NULL},
+ {"line_buffering", (getter)stringio_line_buffering, NULL, NULL},
+ {0}
+};
+
+PyTypeObject PyStringIO_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "_stringio._StringIO", /*tp_name*/
+ "_io.StringIO", /*tp_name*/
sizeof(StringIOObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)stringio_dealloc, /*tp_dealloc*/
@@ -333,51 +746,24 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
- 0, /*tp_doc*/
- 0, /*tp_traverse*/
- 0, /*tp_clear*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
+ stringio_doc, /*tp_doc*/
+ (traverseproc)stringio_traverse, /*tp_traverse*/
+ (inquiry)stringio_clear, /*tp_clear*/
0, /*tp_richcompare*/
- 0, /*tp_weaklistoffset*/
+ offsetof(StringIOObject, weakreflist), /*tp_weaklistoffset*/
0, /*tp_iter*/
- 0, /*tp_iternext*/
+ (iternextfunc)stringio_iternext, /*tp_iternext*/
stringio_methods, /*tp_methods*/
0, /*tp_members*/
- 0, /*tp_getset*/
+ stringio_getset, /*tp_getset*/
0, /*tp_base*/
0, /*tp_dict*/
0, /*tp_descr_get*/
0, /*tp_descr_set*/
- 0, /*tp_dictoffset*/
- 0, /*tp_init*/
+ offsetof(StringIOObject, dict), /*tp_dictoffset*/
+ (initproc)stringio_init, /*tp_init*/
0, /*tp_alloc*/
stringio_new, /*tp_new*/
};
-
-static struct PyModuleDef _stringiomodule = {
- PyModuleDef_HEAD_INIT,
- "_stringio",
- NULL,
- -1,
- NULL,
- NULL,
- NULL,
- NULL,
- NULL
-};
-
-PyMODINIT_FUNC
-PyInit__stringio(void)
-{
- PyObject *m;
-
- if (PyType_Ready(&StringIO_Type) < 0)
- return NULL;
- m = PyModule_Create(&_stringiomodule);
- if (m == NULL)
- return NULL;
- Py_INCREF(&StringIO_Type);
- if (PyModule_AddObject(m, "_StringIO", (PyObject *)&StringIO_Type) < 0)
- return NULL;
- return m;
-}
Copied: python/branches/py3k/Modules/_textio.c (from r68756, /python/branches/io-c/Modules/_textio.c)
==============================================================================
--- /python/branches/io-c/Modules/_textio.c (original)
+++ python/branches/py3k/Modules/_textio.c Wed Mar 4 01:14:51 2009
@@ -1,3 +1,11 @@
+/*
+ An implementation of Text I/O as defined by PEP 3116 - "New I/O"
+
+ Classes defined here: TextIOBase, IncrementalNewlineDecoder, TextIOWrapper.
+
+ Written by Amaury Forgeot d'Arc and Antoine Pitrou
+*/
+
#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
@@ -16,30 +24,128 @@
static PyObject *
_unsupported(const char *message)
{
- PyErr_SetString(PyIOExc_UnsupportedOperation, message);
+ PyErr_SetString(IO_STATE->unsupported_operation, message);
return NULL;
}
+PyDoc_STRVAR(TextIOBase_read_doc,
+ "Read at most n characters from stream.\n"
+ "\n"
+ "Read from underlying buffer until we have n characters or we hit EOF.\n"
+ "If n is negative or omitted, read until EOF.\n"
+ );
+
static PyObject *
TextIOBase_read(PyObject *self, PyObject *args)
{
return _unsupported("read");
}
+PyDoc_STRVAR(TextIOBase_readline_doc,
+ "Read until newline or EOF.\n"
+ "\n"
+ "Returns an empty string if EOF is hit immediately.\n"
+ );
+
+static PyObject *
+TextIOBase_readline(PyObject *self, PyObject *args)
+{
+ return _unsupported("readline");
+}
+
+PyDoc_STRVAR(TextIOBase_write_doc,
+ "Write string to stream.\n"
+ "Returns the number of characters written (which is always equal to\n"
+ "the length of the string).\n"
+ );
+
static PyObject *
TextIOBase_write(PyObject *self, PyObject *args)
{
- return _unsupported("read");
+ return _unsupported("write");
}
+PyDoc_STRVAR(TextIOBase_encoding_doc,
+ "Encoding of the text stream.\n"
+ "\n"
+ "Subclasses should override.\n"
+ );
+
static PyObject *
-TextIOBase_readline(PyObject *self, PyObject *args)
+TextIOBase_encoding_get(PyObject *self, void *context)
{
- return _unsupported("read");
+ Py_RETURN_NONE;
}
-/* XXX properties: encoding, newlines */
-
+PyDoc_STRVAR(TextIOBase_newlines_doc,
+ "Line endings translated so far.\n"
+ "\n"
+ "Only line endings translated during reading are considered.\n"
+ "\n"
+ "Subclasses should override.\n"
+ );
+
+static PyObject *
+TextIOBase_newlines_get(PyObject *self, void *context)
+{
+ Py_RETURN_NONE;
+}
+
+
+static PyMethodDef TextIOBase_methods[] = {
+ {"read", TextIOBase_read, METH_VARARGS, TextIOBase_read_doc},
+ {"readline", TextIOBase_readline, METH_VARARGS, TextIOBase_readline_doc},
+ {"write", TextIOBase_write, METH_VARARGS, TextIOBase_write_doc},
+ {NULL, NULL}
+};
+
+static PyGetSetDef TextIOBase_getset[] = {
+ {"encoding", (getter)TextIOBase_encoding_get, NULL, TextIOBase_encoding_doc},
+ {"newlines", (getter)TextIOBase_newlines_get, NULL, TextIOBase_newlines_doc},
+ {0}
+};
+
+PyTypeObject PyTextIOBase_Type = {
+ PyVarObject_HEAD_INIT(NULL, 0)
+ "_io._TextIOBase", /*tp_name*/
+ 0, /*tp_basicsize*/
+ 0, /*tp_itemsize*/
+ 0, /*tp_dealloc*/
+ 0, /*tp_print*/
+ 0, /*tp_getattr*/
+ 0, /*tp_setattr*/
+ 0, /*tp_compare */
+ 0, /*tp_repr*/
+ 0, /*tp_as_number*/
+ 0, /*tp_as_sequence*/
+ 0, /*tp_as_mapping*/
+ 0, /*tp_hash */
+ 0, /*tp_call*/
+ 0, /*tp_str*/
+ 0, /*tp_getattro*/
+ 0, /*tp_setattro*/
+ 0, /*tp_as_buffer*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ TextIOBase_doc, /* tp_doc */
+ 0, /* tp_traverse */
+ 0, /* tp_clear */
+ 0, /* tp_richcompare */
+ 0, /* tp_weaklistoffset */
+ 0, /* tp_iter */
+ 0, /* tp_iternext */
+ TextIOBase_methods, /* tp_methods */
+ 0, /* tp_members */
+ TextIOBase_getset, /* tp_getset */
+ &PyIOBase_Type, /* tp_base */
+ 0, /* tp_dict */
+ 0, /* tp_descr_get */
+ 0, /* tp_descr_set */
+ 0, /* tp_dictoffset */
+ 0, /* tp_init */
+ 0, /* tp_alloc */
+ 0, /* tp_new */
+};
+
/* IncrementalNewlineDecoder */
@@ -48,7 +154,9 @@
"another incremental decoder, translating \\r\\n and \\r into \\n. It also\n"
"records the types of newlines encountered. When used with\n"
"translate=False, it ensures that the newline sequence is returned in\n"
- "one piece.\n"
+ "one piece. When used with decoder=None, it expects unicode strings as\n"
+ "decode input and translates newlines without first invoking an external\n"
+ "decoder.\n"
);
typedef struct {
@@ -106,12 +214,13 @@
#define SEEN_CRLF 4
#define SEEN_ALL (SEEN_CR | SEEN_LF | SEEN_CRLF)
-static PyObject *
-_IncrementalNewlineDecoder_decode(PyNewLineDecoderObject *self,
- PyObject *input, int final)
+PyObject *
+_PyIncrementalNewlineDecoder_decode(PyObject *_self,
+ PyObject *input, int final)
{
PyObject *output;
Py_ssize_t output_len;
+ PyNewLineDecoderObject *self = (PyNewLineDecoderObject *) _self;
if (self->decoder == NULL) {
PyErr_SetString(PyExc_ValueError,
@@ -120,8 +229,15 @@
}
/* decode input (with the eventual \r from a previous pass) */
- output = PyObject_CallMethodObjArgs(self->decoder, _PyIO_str_decode,
- input, final ? Py_True : Py_False, NULL);
+ if (self->decoder != Py_None) {
+ output = PyObject_CallMethodObjArgs(self->decoder,
+ _PyIO_str_decode, input, final ? Py_True : Py_False, NULL);
+ }
+ else {
+ output = input;
+ Py_INCREF(output);
+ }
+
if (output == NULL)
return NULL;
@@ -177,7 +293,7 @@
Py_UNICODE *in_str;
Py_ssize_t len;
int seennl = self->seennl;
- int only_lf;
+ int only_lf = 0;
in_str = PyUnicode_AS_UNICODE(output);
len = PyUnicode_GET_SIZE(output);
@@ -188,8 +304,16 @@
/* If, up to now, newlines are consistently \n, do a quick check
for the \r *byte* with the libc's optimized memchr.
*/
- only_lf = ((seennl == SEEN_LF)
- && !memchr(in_str, '\r', len * sizeof(Py_UNICODE)));
+ if (seennl == SEEN_LF || seennl == 0) {
+ int has_cr, has_lf;
+ has_lf = (seennl == SEEN_LF) ||
+ (memchr(in_str, '\n', len * sizeof(Py_UNICODE)) != NULL);
+ has_cr = (memchr(in_str, '\r', len * sizeof(Py_UNICODE)) != NULL);
+ if (has_lf && !has_cr) {
+ only_lf = 1;
+ seennl = SEEN_LF;
+ }
+ }
if (!self->translate) {
Py_UNICODE *s, *end;
@@ -301,26 +425,31 @@
if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|i:IncrementalNewlineDecoder",
kwlist, &input, &final))
return NULL;
- return _IncrementalNewlineDecoder_decode(self, input, final);
+ return _PyIncrementalNewlineDecoder_decode((PyObject *) self, input, final);
}
static PyObject *
IncrementalNewlineDecoder_getstate(PyNewLineDecoderObject *self, PyObject *args)
{
- PyObject *state = PyObject_CallMethodObjArgs(self->decoder,
- _PyIO_str_getstate, NULL);
PyObject *buffer;
unsigned PY_LONG_LONG flag;
- if (state == NULL)
- return NULL;
-
- if (!PyArg_Parse(state, "(OK)", &buffer, &flag)) {
+ if (self->decoder != Py_None) {
+ PyObject *state = PyObject_CallMethodObjArgs(self->decoder,
+ _PyIO_str_getstate, NULL);
+ if (state == NULL)
+ return NULL;
+ if (!PyArg_Parse(state, "(OK)", &buffer, &flag)) {
+ Py_DECREF(state);
+ return NULL;
+ }
+ Py_INCREF(buffer);
Py_DECREF(state);
- return NULL;
}
- Py_INCREF(buffer);
- Py_DECREF(state);
+ else {
+ buffer = PyBytes_FromString("");
+ flag = 0;
+ }
flag <<= 1;
if (self->pendingcr)
flag |= 1;
@@ -339,7 +468,11 @@
self->pendingcr = (int) flag & 1;
flag >>= 1;
- return PyObject_CallMethod(self->decoder, "setstate", "((OK))", buffer, flag);
+ if (self->decoder != Py_None)
+ return PyObject_CallMethod(self->decoder,
+ "setstate", "((OK))", buffer, flag);
+ else
+ Py_RETURN_NONE;
}
static PyObject *
@@ -347,7 +480,10 @@
{
self->seennl = 0;
self->pendingcr = 0;
- return PyObject_CallMethod(self->decoder, "reset", NULL);
+ if (self->decoder != Py_None)
+ return PyObject_CallMethodObjArgs(self->decoder, _PyIO_str_reset, NULL);
+ else
+ Py_RETURN_NONE;
}
static PyObject *
@@ -390,7 +526,7 @@
PyTypeObject PyIncrementalNewlineDecoder_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "IncrementalNewlineDecoder", /*tp_name*/
+ "_io.IncrementalNewlineDecoder", /*tp_name*/
sizeof(PyNewLineDecoderObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)IncrementalNewlineDecoder_dealloc, /*tp_dealloc*/
@@ -470,12 +606,12 @@
PyObject *readnl;
PyObject *errors;
const char *writenl; /* utf-8 encoded, NULL stands for \n */
- int line_buffering:1;
- int readuniversal:1;
- int readtranslate:1;
- int writetranslate:1;
- int seekable:1;
- int telling:1;
+ char line_buffering;
+ char readuniversal;
+ char readtranslate;
+ char writetranslate;
+ char seekable;
+ char telling;
/* Specialized encoding func (see below) */
encodefunc_t encodefunc;
@@ -500,6 +636,9 @@
* snapshot point. We use this to reconstruct decoder states in tell().
*/
+ /* Cache raw object if it's a FileIO object */
+ PyObject *raw;
+
PyObject *weakreflist;
PyObject *dict;
} PyTextIOWrapperObject;
@@ -591,11 +730,12 @@
char *kwlist[] = {"buffer", "encoding", "errors",
"newline", "line_buffering",
NULL};
- PyObject *buffer;
+ PyObject *buffer, *raw;
char *encoding = NULL;
char *errors = NULL;
char *newline = NULL;
int line_buffering = 0;
+ _PyIO_State *state = IO_STATE;
PyObject *res;
int r;
@@ -624,27 +764,29 @@
Py_CLEAR(self->pending_bytes);
Py_CLEAR(self->snapshot);
Py_CLEAR(self->errors);
+ Py_CLEAR(self->raw);
self->decoded_chars_used = 0;
self->pending_bytes_count = 0;
self->encodefunc = NULL;
if (encoding == NULL) {
/* Try os.device_encoding(fileno) */
- PyObject *fileno, *os;;
+ PyObject *fileno;
fileno = PyObject_CallMethod(buffer, "fileno", NULL);
- /* Ignore any error */
- /* XXX only AttributeError and UnsupportedOperation */
- if (fileno == NULL)
- PyErr_Clear();
- else {
- os = PyImport_ImportModule("os");
- if (os == NULL) {
- Py_DECREF(fileno);
+ /* Ignore only AttributeError and UnsupportedOperation */
+ if (fileno == NULL) {
+ if (PyErr_ExceptionMatches(PyExc_AttributeError) ||
+ PyErr_ExceptionMatches(state->unsupported_operation)) {
+ PyErr_Clear();
+ }
+ else {
goto error;
}
- self->encoding = PyObject_CallMethod(os, "device_encoding",
+ }
+ else {
+ self->encoding = PyObject_CallMethod(state->os_module,
+ "device_encoding",
"N", fileno);
- Py_DECREF(os);
if (self->encoding == NULL)
goto error;
else if (!PyUnicode_Check(self->encoding))
@@ -652,19 +794,33 @@
}
}
if (encoding == NULL && self->encoding == NULL) {
- /* try locale.getpreferredencoding() */
- PyObject *locale = PyImport_ImportModule("locale");
- if (locale == NULL) {
- PyErr_Clear();
- self->encoding = PyUnicode_FromString("ascii");
+ if (state->locale_module == NULL) {
+ state->locale_module = PyImport_ImportModule("locale");
+ if (state->locale_module == NULL)
+ goto catch_ImportError;
+ else
+ goto use_locale;
}
else {
+ use_locale:
self->encoding = PyObject_CallMethod(
- locale, "getpreferredencoding", NULL);
- Py_DECREF(locale);
- if (self->encoding == NULL)
- goto error;
- if (!PyUnicode_Check(self->encoding))
+ state->locale_module, "getpreferredencoding", NULL);
+ if (self->encoding == NULL) {
+ catch_ImportError:
+ /*
+ Importing locale can raise a ImportError because of
+ _functools, and locale.getpreferredencoding can raise a
+ ImportError if _locale is not available. These will happen
+ during module building.
+ */
+ if (PyErr_ExceptionMatches(PyExc_ImportError)) {
+ PyErr_Clear();
+ self->encoding = PyUnicode_FromString("ascii");
+ }
+ else
+ goto error;
+ }
+ else if (!PyUnicode_Check(self->encoding))
Py_CLEAR(self->encoding);
}
}
@@ -690,8 +846,11 @@
self->readuniversal = (newline == NULL || newline[0] == '\0');
self->line_buffering = line_buffering;
self->readtranslate = (newline == NULL);
- if (newline)
+ if (newline) {
self->readnl = PyUnicode_FromString(newline);
+ if (self->readnl == NULL)
+ return -1;
+ }
self->writetranslate = (newline == NULL || newline[0] != '\0');
if (!self->readuniversal && self->readnl) {
self->writenl = _PyUnicode_AsString(self->readnl);
@@ -765,6 +924,19 @@
self->buffer = buffer;
Py_INCREF(buffer);
+
+ if (Py_TYPE(buffer) == &PyBufferedReader_Type ||
+ Py_TYPE(buffer) == &PyBufferedWriter_Type ||
+ Py_TYPE(buffer) == &PyBufferedRandom_Type) {
+ raw = PyObject_GetAttrString(buffer, "raw");
+ /* Cache the raw FileIO object to speed up 'closed' checks */
+ if (raw == NULL)
+ PyErr_Clear();
+ else if (Py_TYPE(raw) == &PyFileIO_Type)
+ self->raw = raw;
+ else
+ Py_DECREF(raw);
+ }
res = PyObject_CallMethod(buffer, "seekable", NULL);
if (res == NULL)
@@ -779,11 +951,11 @@
return -1;
}
-static void
-TextIOWrapper_dealloc(PyTextIOWrapperObject *self)
+static int
+_TextIOWrapper_clear(PyTextIOWrapperObject *self)
{
- if (_PyIOBase_finalize((PyObject *) self) < 0)
- return;
+ if (self->ok && _PyIOBase_finalize((PyObject *) self) < 0)
+ return -1;
self->ok = 0;
Py_CLEAR(self->buffer);
Py_CLEAR(self->encoding);
@@ -791,31 +963,72 @@
Py_CLEAR(self->decoder);
Py_CLEAR(self->readnl);
Py_CLEAR(self->decoded_chars);
+ Py_CLEAR(self->pending_bytes);
Py_CLEAR(self->snapshot);
+ Py_CLEAR(self->errors);
+ Py_CLEAR(self->raw);
+ return 0;
+}
+
+static void
+TextIOWrapper_dealloc(PyTextIOWrapperObject *self)
+{
+ if (_TextIOWrapper_clear(self) < 0)
+ return;
+ _PyObject_GC_UNTRACK(self);
if (self->weakreflist != NULL)
PyObject_ClearWeakRefs((PyObject *)self);
Py_CLEAR(self->dict);
Py_TYPE(self)->tp_free((PyObject *)self);
}
+static int
+TextIOWrapper_traverse(PyTextIOWrapperObject *self, visitproc visit, void *arg)
+{
+ Py_VISIT(self->buffer);
+ Py_VISIT(self->encoding);
+ Py_VISIT(self->encoder);
+ Py_VISIT(self->decoder);
+ Py_VISIT(self->readnl);
+ Py_VISIT(self->decoded_chars);
+ Py_VISIT(self->pending_bytes);
+ Py_VISIT(self->snapshot);
+ Py_VISIT(self->errors);
+ Py_VISIT(self->raw);
+
+ Py_VISIT(self->dict);
+ return 0;
+}
+
+static int
+TextIOWrapper_clear(PyTextIOWrapperObject *self)
+{
+ if (_TextIOWrapper_clear(self) < 0)
+ return -1;
+ Py_CLEAR(self->dict);
+ return 0;
+}
+
static PyObject *
TextIOWrapper_closed_get(PyTextIOWrapperObject *self, void *context);
-/* This macro takes some shortcuts to make the common case faster. We could
- specialize even more, by detecting that the underlying buffer is a
- BufferedObject. */
+/* This macro takes some shortcuts to make the common case faster. */
#define CHECK_CLOSED(self) \
do { \
int r; \
PyObject *_res; \
if (Py_TYPE(self) == &PyTextIOWrapper_Type) { \
- _res = TextIOWrapper_closed_get(self, NULL); \
- if (_res == NULL) \
- return NULL; \
- r = PyObject_IsTrue(_res); \
- Py_DECREF(_res); \
- if (r < 0) \
- return NULL; \
+ if (self->raw != NULL) \
+ r = _PyFileIO_closed(self->raw); \
+ else { \
+ _res = TextIOWrapper_closed_get(self, NULL); \
+ if (_res == NULL) \
+ return NULL; \
+ r = PyObject_IsTrue(_res); \
+ Py_DECREF(_res); \
+ if (r < 0) \
+ return NULL; \
+ } \
if (r > 0) { \
PyErr_SetString(PyExc_ValueError, \
"I/O operation on closed file."); \
@@ -862,7 +1075,7 @@
if (self->pending_bytes == NULL)
return 0;
- b = _PyBytes_Join(PyBytes_FromStringAndSize(NULL, 0), self->pending_bytes);
+ b = _PyBytes_Join(_PyIO_empty_bytes, self->pending_bytes);
if (b == NULL)
return -1;
ret = PyObject_CallMethodObjArgs(self->buffer,
@@ -1009,28 +1222,6 @@
return chars;
}
-/* Rewind the _decoded_chars buffer. */
-static int
-TextIOWrapper_rewind_decoded_chars(PyTextIOWrapperObject *self, Py_ssize_t n)
-{
- if (self->decoded_chars_used < n) {
- PyErr_SetString(PyExc_ValueError,
- "rewind decoded_chars out of bounds");
- return -1;
- }
- self->decoded_chars_used -= n;
- return self->decoded_chars_used;
-}
-
-static Py_ssize_t
-TextIOWrapper_decoded_chars_length(PyTextIOWrapperObject *self, Py_ssize_t n)
-{
- if (self->decoded_chars == NULL)
- return 0;
- return PyUnicode_GetSize(self->decoded_chars) - self->decoded_chars_used;
-}
-
-
/* Read and decode the next chunk of data from the BufferedReader.
*/
static int
@@ -1088,8 +1279,8 @@
eof = (PyBytes_Size(input_chunk) == 0);
if (Py_TYPE(self->decoder) == &PyIncrementalNewlineDecoder_Type) {
- decoded_chars = _IncrementalNewlineDecoder_decode(
- (PyNewLineDecoderObject *) self->decoder, input_chunk, eof);
+ decoded_chars = _PyIncrementalNewlineDecoder_decode(
+ self->decoder, input_chunk, eof);
}
else {
decoded_chars = PyObject_CallMethodObjArgs(self->decoder,
@@ -1201,8 +1392,7 @@
if (result != NULL && PyList_Append(chunks, result) < 0)
goto fail;
Py_CLEAR(result);
- result = PyUnicode_Join(PyUnicode_FromStringAndSize(NULL, 0),
- chunks);
+ result = PyUnicode_Join(_PyIO_empty_str, chunks);
if (result == NULL)
goto fail;
Py_CLEAR(chunks);
@@ -1234,22 +1424,14 @@
}
}
-/* Finds the first line ending between start and end.
- If not found, returns -1 and sets (*consumed) to the number of characters
- which can be safely put aside before another search.
- If found, returns the index after the line ending and doesn't touch
- (*consumed).
-
- NOTE: `end` must point to the real end of the Py_UNICODE storage,
- that is to the NUL character. Otherwise the function will produce
- incorrect results. */
-static Py_ssize_t
-find_line_ending(PyTextIOWrapperObject *self,
- Py_UNICODE *start, Py_UNICODE *end, Py_ssize_t *consumed)
+Py_ssize_t
+_PyIO_find_line_ending(
+ int translated, int universal, PyObject *readnl,
+ Py_UNICODE *start, Py_UNICODE *end, Py_ssize_t *consumed)
{
Py_ssize_t len = end - start;
- if (self->readtranslate) {
+ if (translated) {
/* Newlines are already translated, only search for \n */
Py_UNICODE *pos = find_control_char(start, end, '\n');
if (pos != NULL)
@@ -1259,7 +1441,7 @@
return -1;
}
}
- else if (self->readuniversal) {
+ else if (universal) {
/* Universal newline search. Find any of \r, \r\n, \n
* The decoder ensures that \r\n are not split in two pieces
*/
@@ -1287,8 +1469,8 @@
}
else {
/* Non-universal mode. */
- Py_ssize_t readnl_len = PyUnicode_GET_SIZE(self->readnl);
- Py_UNICODE *nl = PyUnicode_AS_UNICODE(self->readnl);
+ Py_ssize_t readnl_len = PyUnicode_GET_SIZE(readnl);
+ Py_UNICODE *nl = PyUnicode_AS_UNICODE(readnl);
if (readnl_len == 1) {
Py_UNICODE *pos = find_control_char(start, end, nl[0]);
if (pos != NULL)
@@ -1381,7 +1563,9 @@
ptr = PyUnicode_AS_UNICODE(line);
line_len = PyUnicode_GET_SIZE(line);
- endpos = find_line_ending(self, ptr + start, ptr + line_len, &consumed);
+ endpos = _PyIO_find_line_ending(
+ self->readtranslate, self->readuniversal, self->readnl,
+ ptr + start, ptr + line_len, &consumed);
if (endpos >= 0) {
endpos += start;
if (limit >= 0 && (endpos - start) + chunked >= limit)
@@ -1460,7 +1644,7 @@
if (line != NULL && PyList_Append(chunks, line) < 0)
goto error;
Py_CLEAR(line);
- line = PyUnicode_Join(PyUnicode_FromStringAndSize(NULL, 0), chunks);
+ line = PyUnicode_Join(_PyIO_empty_str, chunks);
if (line == NULL)
goto error;
Py_DECREF(chunks);
@@ -1588,7 +1772,7 @@
utf-16, that we are expecting a BOM).
*/
if (cookie->start_pos == 0 && cookie->dec_flags == 0)
- res = PyObject_CallMethod(self->decoder, "reset", NULL);
+ res = PyObject_CallMethodObjArgs(self->decoder, _PyIO_str_reset, NULL);
else
res = PyObject_CallMethod(self->decoder, "setstate",
"((yi))", "", cookie->dec_flags);
@@ -1601,7 +1785,7 @@
static PyObject *
TextIOWrapper_seek(PyTextIOWrapperObject *self, PyObject *args)
{
- PyObject *cookieObj;
+ PyObject *cookieObj, *posobj;
CookieStruct cookie;
int whence = 0;
static PyObject *zero = NULL;
@@ -1618,10 +1802,10 @@
if (!PyArg_ParseTuple(args, "O|i:seek", &cookieObj, &whence))
return NULL;
- Py_INCREF(cookieObj);
-
CHECK_CLOSED(self);
+ Py_INCREF(cookieObj);
+
if (!self->seekable) {
PyErr_SetString(PyExc_IOError,
"underlying stream is not seekable");
@@ -1676,6 +1860,7 @@
}
res = PyObject_CallMethod(self->buffer, "seek", "ii", 0, 2);
+ Py_XDECREF(cookieObj);
return res;
}
else if (whence != 0) {
@@ -1694,7 +1879,7 @@
goto fail;
}
- res = PyObject_CallMethod((PyObject *)self, "flush", NULL);
+ res = PyObject_CallMethodObjArgs((PyObject *)self, _PyIO_str_flush, NULL);
if (res == NULL)
goto fail;
Py_DECREF(res);
@@ -1706,8 +1891,12 @@
goto fail;
/* Seek back to the safe start point. */
- res = PyObject_CallMethod(self->buffer, "seek",
- "L", (PY_LONG_LONG)cookie.start_pos);
+ posobj = PyLong_FromOff_t(cookie.start_pos);
+ if (posobj == NULL)
+ goto fail;
+ res = PyObject_CallMethodObjArgs(self->buffer,
+ _PyIO_str_seek, posobj, NULL);
+ Py_DECREF(posobj);
if (res == NULL)
goto fail;
Py_DECREF(res);
@@ -1719,9 +1908,6 @@
if (self->decoder) {
if (_TextIOWrapper_decoder_setstate(self, &cookie) < 0)
goto fail;
- self->snapshot = Py_BuildValue("iy", cookie.dec_flags, "");
- if (self->snapshot == NULL)
- goto fail;
}
if (cookie.chars_to_skip) {
@@ -1735,14 +1921,14 @@
assert (PyBytes_Check(input_chunk));
- self->snapshot = Py_BuildValue("iO", cookie.dec_flags, input_chunk);
+ self->snapshot = Py_BuildValue("iN", cookie.dec_flags, input_chunk);
if (self->snapshot == NULL) {
Py_DECREF(input_chunk);
goto fail;
}
decoded = PyObject_CallMethod(self->decoder, "decode",
- "Ni", input_chunk, (int)cookie.need_eof);
+ "Oi", input_chunk, (int)cookie.need_eof);
if (decoded == NULL)
goto fail;
@@ -1756,6 +1942,11 @@
}
self->decoded_chars_used = cookie.chars_to_skip;
}
+ else {
+ self->snapshot = Py_BuildValue("iy", cookie.dec_flags, "");
+ if (self->snapshot == NULL)
+ goto fail;
+ }
return cookieObj;
fail:
@@ -1896,6 +2087,7 @@
goto fail;
assert (PyUnicode_Check(decoded));
chars_decoded += PyUnicode_GET_SIZE(decoded);
+ Py_DECREF(decoded);
cookie.need_eof = 1;
if (chars_decoded < chars_to_skip) {
@@ -2139,6 +2331,7 @@
static PyMemberDef TextIOWrapper_members[] = {
{"encoding", T_OBJECT, offsetof(PyTextIOWrapperObject, encoding), READONLY},
{"buffer", T_OBJECT, offsetof(PyTextIOWrapperObject, buffer), READONLY},
+ {"line_buffering", T_BOOL, offsetof(PyTextIOWrapperObject, line_buffering), READONLY},
{NULL}
};
@@ -2155,7 +2348,7 @@
PyTypeObject PyTextIOWrapper_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- "TextIOWrapper", /*tp_name*/
+ "_io.TextIOWrapper", /*tp_name*/
sizeof(PyTextIOWrapperObject), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor)TextIOWrapper_dealloc, /*tp_dealloc*/
@@ -2173,10 +2366,11 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE
+ | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
TextIOWrapper_doc, /* tp_doc */
- 0, /* tp_traverse */
- 0, /* tp_clear */
+ (traverseproc)TextIOWrapper_traverse, /* tp_traverse */
+ (inquiry)TextIOWrapper_clear, /* tp_clear */
0, /* tp_richcompare */
offsetof(PyTextIOWrapperObject, weakreflist), /*tp_weaklistoffset*/
0, /* tp_iter */
Copied: python/branches/py3k/Modules/io.c (from r68756, /python/branches/io-c/Modules/io.c)
==============================================================================
--- /python/branches/io-c/Modules/io.c (original)
+++ python/branches/py3k/Modules/io.c Wed Mar 4 01:14:51 2009
@@ -1,3 +1,12 @@
+/*
+ An implementation of the new I/O lib as defined by PEP 3116 - "New I/O"
+
+ Classes defined here: UnsupportedOperation, BlockingIOError.
+ Functions defined here: open().
+
+ Mostly written by Amaury Forgeot d'Arc
+*/
+
#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include "structmember.h"
@@ -11,7 +20,6 @@
#include <sys/stat.h>
#endif /* HAVE_SYS_STAT_H */
-PyObject *PyIOExc_UnsupportedOperation;
/* Various interned strings */
@@ -24,11 +32,13 @@
PyObject *_PyIO_str_getstate;
PyObject *_PyIO_str_isatty;
PyObject *_PyIO_str_newlines;
+PyObject *_PyIO_str_nl;
PyObject *_PyIO_str_read;
PyObject *_PyIO_str_read1;
PyObject *_PyIO_str_readable;
PyObject *_PyIO_str_readinto;
PyObject *_PyIO_str_readline;
+PyObject *_PyIO_str_reset;
PyObject *_PyIO_str_seek;
PyObject *_PyIO_str_seekable;
PyObject *_PyIO_str_tell;
@@ -36,6 +46,9 @@
PyObject *_PyIO_str_writable;
PyObject *_PyIO_str_write;
+PyObject *_PyIO_empty_str;
+PyObject *_PyIO_empty_bytes;
+
PyDoc_STRVAR(module_doc,
"The io module provides the Python interfaces to stream handling. The\n"
@@ -82,41 +95,29 @@
BlockingIOError_init(PyBlockingIOErrorObject *self, PyObject *args,
PyObject *kwds)
{
- PyObject *myerrno = NULL, *strerror = NULL, *written;
+ PyObject *myerrno = NULL, *strerror = NULL;
PyObject *baseargs = NULL;
+ Py_ssize_t written = 0;
assert(PyTuple_Check(args));
- if (PyTuple_GET_SIZE(args) <= 1 || PyTuple_GET_SIZE(args) > 3)
- return 0;
+ self->written = 0;
+ if (!PyArg_ParseTuple(args, "OO|n:BlockingIOError",
+ &myerrno, &strerror, &written))
+ return -1;
- baseargs = PyTuple_GetSlice(args, 0, 2);
+ baseargs = PyTuple_Pack(2, myerrno, strerror);
if (baseargs == NULL)
return -1;
-
+ /* This will take care of initializing of myerrno and strerror members */
if (((PyTypeObject *)PyExc_IOError)->tp_init(
(PyObject *)self, baseargs, kwds) == -1) {
Py_DECREF(baseargs);
return -1;
}
-
Py_DECREF(baseargs);
- if (!PyArg_UnpackTuple(args, "BlockingIOError", 2, 3,
- &myerrno, &strerror, &written)) {
- return -1;
- }
-
- Py_INCREF(myerrno);
- self->myerrno = myerrno;
-
- Py_INCREF(strerror);
- self->strerror = strerror;
-
- self->written = PyNumber_AsSsize_t(written, PyExc_ValueError);
- if(self->written == -1 && PyErr_Occurred())
- return -1;
-
+ self->written = written;
return 0;
}
@@ -125,7 +126,6 @@
{NULL} /* Sentinel */
};
-
static PyTypeObject _PyExc_BlockingIOError = {
PyVarObject_HEAD_INIT(NULL, 0)
"BlockingIOError", /*tp_name*/
@@ -146,8 +146,9 @@
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
- Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /*tp_flags*/
- PyDoc_STR("Exception raised when I/O would block on a non-blocking I/O stream"), /* tp_doc */
+ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
+ PyDoc_STR("Exception raised when I/O would block "
+ "on a non-blocking I/O stream"), /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
@@ -558,6 +559,36 @@
return result;
}
+static int
+iomodule_traverse(PyObject *mod, visitproc visit, void *arg) {
+ _PyIO_State *state = IO_MOD_STATE(mod);
+ if (!state->initialized)
+ return 0;
+ Py_VISIT(state->os_module);
+ if (state->locale_module != NULL) {
+ Py_VISIT(state->locale_module);
+ }
+ Py_VISIT(state->unsupported_operation);
+ return 0;
+}
+
+static int
+iomodule_clear(PyObject *mod) {
+ _PyIO_State *state = IO_MOD_STATE(mod);
+ if (!state->initialized)
+ return 0;
+ Py_CLEAR(state->os_module);
+ if (state->locale_module != NULL)
+ Py_CLEAR(state->locale_module);
+ Py_CLEAR(state->unsupported_operation);
+ return 0;
+}
+
+static void
+iomodule_free(PyObject *mod) {
+ iomodule_clear(mod);
+}
+
/*
* Module definition
*/
@@ -567,116 +598,104 @@
{NULL, NULL}
};
-static struct PyModuleDef iomodule = {
+struct PyModuleDef _PyIO_Module = {
PyModuleDef_HEAD_INIT,
"io",
module_doc,
- -1,
+ sizeof(_PyIO_State),
module_methods,
NULL,
- NULL,
- NULL,
- NULL
+ iomodule_traverse,
+ iomodule_clear,
+ (freefunc)iomodule_free,
};
PyMODINIT_FUNC
PyInit__io(void)
{
- PyObject *m = PyModule_Create(&iomodule);
- PyTypeObject *base;
+ PyObject *m = PyModule_Create(&_PyIO_Module);
+ _PyIO_State *state = NULL;
if (m == NULL)
+ return NULL;
+ state = IO_MOD_STATE(m);
+ state->initialized = 0;
+
+ /* put os in the module state */
+ state->os_module = PyImport_ImportModule("os");
+ if (state->os_module == NULL)
+ goto fail;
+
+#define ADD_TYPE(type, name) \
+ if (PyType_Ready(type) < 0) \
+ goto fail; \
+ Py_INCREF(type); \
+ if (PyModule_AddObject(m, name, (PyObject *)type) < 0) { \
+ Py_DECREF(type); \
+ goto fail; \
+ }
+
+ /* DEFAULT_BUFFER_SIZE */
+ if (PyModule_AddIntMacro(m, DEFAULT_BUFFER_SIZE) < 0)
goto fail;
/* UnsupportedOperation inherits from ValueError and IOError */
- PyIOExc_UnsupportedOperation = PyObject_CallFunction(
+ state->unsupported_operation = PyObject_CallFunction(
(PyObject *)&PyType_Type, "s(OO){}",
"UnsupportedOperation", PyExc_ValueError, PyExc_IOError);
- if (PyIOExc_UnsupportedOperation == NULL)
+ if (state->unsupported_operation == NULL)
goto fail;
- PyModule_AddObject(m, "UnsupportedOperation",
- PyIOExc_UnsupportedOperation);
-
- /* BlockingIOError */
- base = (PyTypeObject *)PyExc_IOError;
- _PyExc_BlockingIOError.tp_base = base;
- if (PyType_Ready(&_PyExc_BlockingIOError) < 0)
+ Py_INCREF(state->unsupported_operation);
+ if (PyModule_AddObject(m, "UnsupportedOperation",
+ state->unsupported_operation) < 0)
goto fail;
- Py_INCREF(&_PyExc_BlockingIOError);
- PyModule_AddObject(m, "BlockingIOError",
- (PyObject *)&_PyExc_BlockingIOError);
- if (PyType_Ready(&PyIOBase_Type) < 0)
- goto fail;
- Py_INCREF(&PyIOBase_Type);
- PyModule_AddObject(m, "IOBase",
- (PyObject *)&PyIOBase_Type);
+ /* BlockingIOError */
+ _PyExc_BlockingIOError.tp_base = (PyTypeObject *) PyExc_IOError;
+ ADD_TYPE(&_PyExc_BlockingIOError, "BlockingIOError");
- if (PyType_Ready(&PyRawIOBase_Type) < 0)
- goto fail;
- Py_INCREF(&PyRawIOBase_Type);
- PyModule_AddObject(m, "RawIOBase",
- (PyObject *)&PyRawIOBase_Type);
+ /* Concrete base types of the IO ABCs.
+ (the ABCs themselves are declared through inheritance in io.py)
+ */
+ ADD_TYPE(&PyIOBase_Type, "_IOBase");
+ ADD_TYPE(&PyRawIOBase_Type, "_RawIOBase");
+ ADD_TYPE(&PyBufferedIOBase_Type, "_BufferedIOBase");
+ ADD_TYPE(&PyTextIOBase_Type, "_TextIOBase");
+ /* Implementation of concrete IO objects. */
/* FileIO */
PyFileIO_Type.tp_base = &PyRawIOBase_Type;
- if (PyType_Ready(&PyFileIO_Type) < 0)
- goto fail;
- Py_INCREF(&PyFileIO_Type);
- PyModule_AddObject(m, "FileIO", (PyObject *) &PyFileIO_Type);
-
- /* BufferedIOBase */
- if (PyType_Ready(&PyBufferedIOBase_Type) < 0)
- goto fail;
- Py_INCREF(&PyBufferedIOBase_Type);
- PyModule_AddObject(m, "BufferedIOBase", (PyObject *) &PyBufferedIOBase_Type);
+ ADD_TYPE(&PyFileIO_Type, "FileIO");
/* BytesIO */
PyBytesIO_Type.tp_base = &PyBufferedIOBase_Type;
- if (PyType_Ready(&PyBytesIO_Type) < 0)
- goto fail;
- Py_INCREF(&PyBytesIO_Type);
- PyModule_AddObject(m, "BytesIO", (PyObject *) &PyBytesIO_Type);
+ ADD_TYPE(&PyBytesIO_Type, "BytesIO");
+
+ /* StringIO */
+ PyStringIO_Type.tp_base = &PyTextIOBase_Type;
+ ADD_TYPE(&PyStringIO_Type, "StringIO");
/* BufferedReader */
PyBufferedReader_Type.tp_base = &PyBufferedIOBase_Type;
- if (PyType_Ready(&PyBufferedReader_Type) < 0)
- goto fail;
- Py_INCREF(&PyBufferedReader_Type);
- PyModule_AddObject(m, "BufferedReader", (PyObject *) &PyBufferedReader_Type);
+ ADD_TYPE(&PyBufferedReader_Type, "BufferedReader");
/* BufferedWriter */
PyBufferedWriter_Type.tp_base = &PyBufferedIOBase_Type;
- if (PyType_Ready(&PyBufferedWriter_Type) < 0)
- goto fail;
- Py_INCREF(&PyBufferedWriter_Type);
- PyModule_AddObject(m, "BufferedWriter", (PyObject *) &PyBufferedWriter_Type);
+ ADD_TYPE(&PyBufferedWriter_Type, "BufferedWriter");
/* BufferedRWPair */
PyBufferedRWPair_Type.tp_base = &PyBufferedIOBase_Type;
- if (PyType_Ready(&PyBufferedRWPair_Type) < 0)
- goto fail;
- Py_INCREF(&PyBufferedRWPair_Type);
- PyModule_AddObject(m, "BufferedRWPair", (PyObject *) &PyBufferedRWPair_Type);
+ ADD_TYPE(&PyBufferedRWPair_Type, "BufferedRWPair");
/* BufferedRandom */
PyBufferedRandom_Type.tp_base = &PyBufferedIOBase_Type;
- if (PyType_Ready(&PyBufferedRandom_Type) < 0)
- goto fail;
- Py_INCREF(&PyBufferedRandom_Type);
- PyModule_AddObject(m, "BufferedRandom", (PyObject *) &PyBufferedRandom_Type);
+ ADD_TYPE(&PyBufferedRandom_Type, "BufferedRandom");
/* TextIOWrapper */
- PyTextIOWrapper_Type.tp_base = &PyIOBase_Type;
- if (PyType_Ready(&PyTextIOWrapper_Type) < 0)
- goto fail;
- Py_INCREF(&PyTextIOWrapper_Type);
- PyModule_AddObject(m, "TextIOWrapper", (PyObject *) &PyTextIOWrapper_Type);
+ PyTextIOWrapper_Type.tp_base = &PyTextIOBase_Type;
+ ADD_TYPE(&PyTextIOWrapper_Type, "TextIOWrapper");
- /* TextIOWrapper */
- if (PyType_Ready(&PyIncrementalNewlineDecoder_Type) < 0)
- goto fail;
- Py_INCREF(&PyIncrementalNewlineDecoder_Type);
- PyModule_AddObject(m, "IncrementalNewlineDecoder", (PyObject *) &PyIncrementalNewlineDecoder_Type);
+ /* IncrementalNewlineDecoder */
+ ADD_TYPE(&PyIncrementalNewlineDecoder_Type, "IncrementalNewlineDecoder");
/* Interned strings */
if (!(_PyIO_str_close = PyUnicode_InternFromString("close")))
@@ -697,6 +716,8 @@
goto fail;
if (!(_PyIO_str_newlines = PyUnicode_InternFromString("newlines")))
goto fail;
+ if (!(_PyIO_str_nl = PyUnicode_InternFromString("\n")))
+ goto fail;
if (!(_PyIO_str_read = PyUnicode_InternFromString("read")))
goto fail;
if (!(_PyIO_str_read1 = PyUnicode_InternFromString("read1")))
@@ -707,6 +728,8 @@
goto fail;
if (!(_PyIO_str_readline = PyUnicode_InternFromString("readline")))
goto fail;
+ if (!(_PyIO_str_reset = PyUnicode_InternFromString("reset")))
+ goto fail;
if (!(_PyIO_str_seek = PyUnicode_InternFromString("seek")))
goto fail;
if (!(_PyIO_str_seekable = PyUnicode_InternFromString("seekable")))
@@ -719,10 +742,19 @@
goto fail;
if (!(_PyIO_str_writable = PyUnicode_InternFromString("writable")))
goto fail;
+
+ if (!(_PyIO_empty_str = PyUnicode_FromStringAndSize(NULL, 0)))
+ goto fail;
+ if (!(_PyIO_empty_bytes = PyBytes_FromStringAndSize(NULL, 0)))
+ goto fail;
+
+ state->initialized = 1;
return m;
fail:
- Py_XDECREF(m);
+ Py_XDECREF(state->os_module);
+ Py_XDECREF(state->unsupported_operation);
+ Py_DECREF(m);
return NULL;
}
Modified: python/branches/py3k/PC/VC6/pythoncore.dsp
==============================================================================
--- python/branches/py3k/PC/VC6/pythoncore.dsp (original)
+++ python/branches/py3k/PC/VC6/pythoncore.dsp Wed Mar 4 01:14:51 2009
@@ -97,6 +97,10 @@
# End Source File
# Begin Source File
+SOURCE=..\..\Modules\_bufferedio.c
+# End Source File
+# Begin Source File
+
SOURCE=..\..\Modules\_bytesio.c
# End Source File
# Begin Source File
@@ -149,6 +153,10 @@
# End Source File
# Begin Source File
+SOURCE=..\..\Modules\_iobase.c
+# End Source File
+# Begin Source File
+
SOURCE=..\..\Modules\_json.c
# End Source File
# Begin Source File
@@ -185,6 +193,10 @@
# End Source File
# Begin Source File
+SOURCE=..\..\Modules\_textio.c
+# End Source File
+# Begin Source File
+
SOURCE=..\..\Modules\_threadmodule.c
# End Source File
# Begin Source File
@@ -463,6 +475,10 @@
# End Source File
# Begin Source File
+SOURCE=..\..\Modules\io.c
+# End Source File
+# Begin Source File
+
SOURCE=..\..\Objects\iterobject.c
# End Source File
# Begin Source File
Modified: python/branches/py3k/PC/config.c
==============================================================================
--- python/branches/py3k/PC/config.c (original)
+++ python/branches/py3k/PC/config.c Wed Mar 4 01:14:51 2009
@@ -58,9 +58,7 @@
extern PyObject* PyInit__subprocess(void);
extern PyObject* PyInit__lsprof(void);
extern PyObject* PyInit__ast(void);
-extern PyObject* PyInit__fileio(void);
-extern PyObject* PyInit__bytesio(void);
-extern PyObject* PyInit__stringio(void);
+extern PyObject* PyInit__io(void);
extern PyObject* PyInit__pickle(void);
extern PyObject* PyInit_atexit(void);
extern PyObject* _PyWarnings_Init(void);
@@ -150,9 +148,7 @@
{"sys", NULL},
{"_warnings", _PyWarnings_Init},
- {"_fileio", PyInit__fileio},
- {"_bytesio", PyInit__bytesio},
- {"_stringio", PyInit__stringio},
+ {"_io", PyInit__io},
{"_pickle", PyInit__pickle},
{"atexit", PyInit_atexit},
Modified: python/branches/py3k/PCbuild/pythoncore.vcproj
==============================================================================
--- python/branches/py3k/PCbuild/pythoncore.vcproj (original)
+++ python/branches/py3k/PCbuild/pythoncore.vcproj Wed Mar 4 01:14:51 2009
@@ -975,6 +975,10 @@
>
</File>
<File
+ RelativePath="..\Modules\_bufferedio.c"
+ >
+ </File>
+ <File
RelativePath="..\Modules\_codecsmodule.c"
>
</File>
@@ -1003,6 +1007,14 @@
>
</File>
<File
+ RelativePath="..\Modules\_iobase.c"
+ >
+ </File>
+ <File
+ RelativePath="..\Modules\_iomodule.h"
+ >
+ </File>
+ <File
RelativePath="..\Modules\_json.c"
>
</File>
@@ -1035,6 +1047,10 @@
>
</File>
<File
+ RelativePath="..\Modules\_textio.c"
+ >
+ </File>
+ <File
RelativePath="..\Modules\_weakref.c"
>
</File>
@@ -1075,6 +1091,10 @@
>
</File>
<File
+ RelativePath="..\Modules\io.c"
+ >
+ </File>
+ <File
RelativePath="..\Modules\main.c"
>
</File>
Modified: python/branches/py3k/Python/pythonrun.c
==============================================================================
--- python/branches/py3k/Python/pythonrun.c (original)
+++ python/branches/py3k/Python/pythonrun.c Wed Mar 4 01:14:51 2009
@@ -769,7 +769,7 @@
}
text = PyUnicode_FromString(name);
- if (text == NULL || PyObject_SetAttrString(raw, "_name", text) < 0)
+ if (text == NULL || PyObject_SetAttrString(raw, "name", text) < 0)
goto error;
res = PyObject_CallMethod(raw, "isatty", "");
if (res == NULL)
Modified: python/branches/py3k/setup.py
==============================================================================
--- python/branches/py3k/setup.py (original)
+++ python/branches/py3k/setup.py Wed Mar 4 01:14:51 2009
@@ -982,8 +982,6 @@
# Thomas Heller's _ctypes module
self.detect_ctypes(inc_dirs, lib_dirs)
- # _fileio -- supposedly cross platform
- exts.append(Extension('_fileio', ['_fileio.c']))
# Richard Oudkerk's multiprocessing module
if platform == 'win32': # Windows
macros = dict()
More information about the Python-checkins
mailing list