NamedTemporaryFile and context managers
In issue14243 [1] there are two issues being tracked: - the difference in opening shared files between posix and Windows - the behavior of closing the underlying file in the middle of NamedTemporaryFile's context management I'd like to address and get feedback on the context management issue. ```python from tempfile import NamedTemporaryFile with NamedTemporaryFile() as fp: fp.write(b'some data') fp = open(fp.name()) data = fp.read() assert data == 'some_data' ``` Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp file for reading while it is still open for writing). This would look like: ```python from tempfile import NamedTemporaryFile with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read() assert data == 'some_data' ``` The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold: 1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the file-management abilities in today's software that seems like a minor annoyance at most. The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one would call close() and then stay inside the CM, outside of testing, I cannot fathom. [3] So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ? -- ~Ethan~ [1] https://bugs.python.org/issue14243 [2] plus, the `.open()` doesn't yet exist [3] feel free to educate me :-)
On Thu, 8 Apr 2021 13:31:26 -0700 Ethan Furman <ethan@stoneleaf.us> wrote:
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
Instead, you could add a dedicated `.reopen()`? Regards Antoine.
On 4/8/21 1:43 PM, Antoine Pitrou wrote:
On Thu, 8 Apr 2021 13:31:26 -0700 Ethan Furman <ethan@stoneleaf.us> wrote:
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
Instead, you could add a dedicated `.reopen()`?
The main hurdle is that on Windows we let the OS manage the lifetime of the file, which means that it is deleted as soon as it is closed. We would need to remove that branch and treat all NamedTemporaryFiles the same. We could add reopen(), but since close() is already there... although I do like the name of `reopen`. -- ~Ethan~
Well this works: from tempfile import NamedTemporaryFile import os with NamedTemporaryFile(delete=False) as fp: fp.write(b'some data') fp.close() with open(fp.name, 'rb') as fp2: data = fp2.read() os.remove(fp.name) assert data == b'some data' Of course it is theoretically possible that another app will do something to the temporary file between the "fp.close()" and the "open" on the next line, or between closing fp2 and the "os.remove". Is this a concern? Rob Cliffe On 08/04/2021 22:42, Ethan Furman wrote:
On 4/8/21 1:43 PM, Antoine Pitrou wrote:
On Thu, 8 Apr 2021 13:31:26 -0700 Ethan Furman <ethan@stoneleaf.us> wrote:
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
Instead, you could add a dedicated `.reopen()`?
The main hurdle is that on Windows we let the OS manage the lifetime of the file, which means that it is deleted as soon as it is closed. We would need to remove that branch and treat all NamedTemporaryFiles the same.
We could add reopen(), but since close() is already there... although I do like the name of `reopen`.
-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2DGIFUS6... Code of Conduct: http://python.org/psf/codeofconduct/
On 4/8/2021 4:43 PM, Antoine Pitrou wrote:
On Thu, 8 Apr 2021 13:31:26 -0700 Ethan Furman <ethan@stoneleaf.us> wrote:
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
Instead, you could add a dedicated `.reopen()`?
I think capturing the intent is important, rather than just putting .close() followed by .open(). Maybe a name that embodies "reopen for reading without deleting" would make the intent clear? Maybe .reopen() already captures that. Anyway, +1 for a method that combines the close/open. Eric
On 8 Apr 2021, at 22:31, Ethan Furman <ethan@stoneleaf.us> wrote:
In issue14243 [1] there are two issues being tracked:
- the difference in opening shared files between posix and Windows - the behavior of closing the underlying file in the middle of NamedTemporaryFile's context management
I'd like to address and get feedback on the context management issue.
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data’) fp.flush(). # needed to ensure data is actually in the file ;-) fp = open(fp.name()) data = fp.read()
assert data == 'some_data' ```
I generally use a slightly different pattern for this: with NamedTemporaryFile() as fp: fp.write(b'some data’) fp.flush() fp.seek(0) data = fp.read() That is, reuse the same “fp” and just reset the stream. An advantage of this approach is that you don’t need a named temporary file for this (and could even use a spooled one). That said, I at times use this pattern with a named temporary file with a quick self-test for the file contents before handing of the file name to an external proces.
Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp file for reading while it is still open for writing). This would look like:
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
I’ve never had the need for such an API, but must say that I barely use Windows and hence have not run into the “cannot open file for reading what it is still open for writing” issue.
A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the file-management abilities in today's software that seems like a minor annoyance at most.
The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one would call close() and then stay inside the CM, outside of testing, I cannot fathom. [3]
So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ?
-- ~Ethan~
Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 08.04.2021 23:31, Ethan Furman wrote:
In issue14243 [1] there are two issues being tracked:
- the difference in opening shared files between posix and Windows - the behavior of closing the underlying file in the middle of NamedTemporaryFile's context management
I'd like to address and get feedback on the context management issue.
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp = open(fp.name()) data = fp.read()
assert data == 'some_data' ```
Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp file for reading while it is still open for writing). This would look like:
What's the problem with `NamedTemporaryFile(mode='w+b')`? (it's actually the default!) You can both read and write without reopening. If you don't want to seek() between reading and writing, you can dup() the descriptor and wrap it with another file object: `fp2=open(os.dup(fp.file.fileno()))`
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read()
assert data == 'some_data' ```
The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold:
1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile
A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the file-management abilities in today's software that seems like a minor annoyance at most.
The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one would call close() and then stay inside the CM, outside of testing, I cannot fathom. [3]
So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ?
-- ~Ethan~
[1] https://bugs.python.org/issue14243 [2] plus, the `.open()` doesn't yet exist [3] feel free to educate me :-) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DZNEIRHZ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Regards, Ivan
08.04.21 23:31, Ethan Furman пише:
In issue14243 [1] there are two issues being tracked:
- the difference in opening shared files between posix and Windows - the behavior of closing the underlying file in the middle of NamedTemporaryFile's context management
I'd like to address and get feedback on the context management issue.
```python from tempfile import NamedTemporaryFile
with NamedTemporaryFile() as fp: fp.write(b'some data') fp = open(fp.name()) data = fp.read()
assert data == 'some_data' ```
These issues are usually solved by using TemporaryDirectory: with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: fp.write(b'some data') with open(filename, 'rb') as fp: data = fp.read() What if make NamedTemporaryFile a wrapper around TemporaryDirectory and always create a new temporary directory for file? @contextmanager def NamedTemporaryFile(): with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: yield fp
participants (7)
-
Antoine Pitrou -
Eric V. Smith -
Ethan Furman -
Ivan Pozdeev -
Rob Cliffe -
Ronald Oussoren -
Serhiy Storchaka