Pickle to/from filename or path
Currently with pickle you have to "open" the file separately from the pickle operation, generally using a "with" clause, then provide the file object to one of the pickle.dump or pickle.load. This adds quite a bit of boilerplate for simply saving a file I think it would be nice if pickle.dump and pickle.load would also accept a filename or path. The functions would automatically open the file in the correct way, save/load the object, then close the file. This would reduce the amount of code needed to save a single object by at least half, and would eliminate the incentive to use unsafe (I think?) approaches like opening within the dump or load, which the wiki still recommends [1]. So something like: with open('myfile.p', 'wb') as f: pickle.dump(myobj, f) Would be: pickle.dump(myobj, 'myfile.p') Sorry if this has already been discussed, but I searched the mailing list history and couldn't find it. [1] https://wiki.python.org/moin/UsingPickle
On Fri, 7 Feb 2020 13:03:52 -0500
Todd
Currently with pickle you have to "open" the file separately from the pickle operation, generally using a "with" clause, then provide the file object to one of the pickle.dump or pickle.load. This adds quite a bit of boilerplate for simply saving a file
What you call "quite a bit of boilerplate" is just an additional line of code. If you are frequently being bothered by this (I'm curious what the context is?), it's easy to add the desired helper function to your own library. In general, I don't think adding tons of trivial convenience helpers is good API design, it ends up making the API more difficult to discover and digest. Regards Antoine.
07.02.20 20:41, Antoine Pitrou пише:
What you call "quite a bit of boilerplate" is just an additional line of code. If you are frequently being bothered by this (I'm curious what the context is?), it's easy to add the desired helper function to your own library.
In general, I don't think adding tons of trivial convenience helpers is good API design, it ends up making the API more difficult to discover and digest.
Concur with Antoine.
Frankly, I’ve often wanted this for other things that share a similar API, JSON comes to mind. Loading from/saving to a file is a pretty common thing— nice to make it easy. And JSON at least has *s versions for strings, so no confusion there. But it’s been this way a LONG time, and with a general trend of Python becoming more of a systems that scripting language, I don’t see it changing. -CHB PS: you don’t need the context manager — at least in cPython — you can put the open call right in the call to load() -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Feb 7, 2020, at 10:21, Todd
wrote: I think it would be nice if pickle.dump and pickle.load would also accept a filename or path.
Pickle intentionally has the same API as marshal, and so do a number of other modules—not just in the stdlib, but popular third party modules. So, I don’t think you’d want to change pickle to be different from everything else, but it might be a major process to get everything changed. Also, while there is code in Python that does this “file object or path” thing, I think it’s mostly older code, and not something to encourage going forward. It can be a confusing API, since there’s _other_ code that does the “file object or string” API (not too much of a problem for dump, but think about load). Maybe a keyword-only parameter would be better? Or even a third function: load for file, loads for string, loadp or load_path for path? Or maybe it would be better to have a generic wrapper that can take any file-using function and give you a path-using function: def opener(func, mode='r') @wraps(func) def wrapper(*args, **kw): with open(args[-1], mode) as f: return func(*args[:-1], f, **kwargs) This seems a little magical to have in the stdlib (especially since obviously not every function that wants a file wants it as the last positional argument—in a different language I’d probably make this take the first argument and then wrap load in opener but dump in flip opener flip…), but might be worth putting in your own toolkit.
On Fri, Feb 7, 2020, at 13:03, Todd wrote:
approaches like opening within the dump or load, which the wiki still recommends [1].
So something like:
with open('myfile.p', 'wb') as f: pickle.dump(myobj, f)
Would be:
pickle.dump(myobj, 'myfile.p')
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))? Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
On Sun, Feb 9, 2020 at 8:04 AM Random832
On Fri, Feb 7, 2020, at 13:03, Todd wrote:
approaches like opening within the dump or load, which the wiki still recommends [1].
So something like:
with open('myfile.p', 'wb') as f: pickle.dump(myobj, f)
Would be:
pickle.dump(myobj, 'myfile.p')
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))?
Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
Exactly how much code would be wrapped in the 'with' block? ChrisA
On Feb 8, 2020, at 13:25, Chris Angelico
On Sun, Feb 9, 2020 at 8:04 AM Random832
wrote: On Fri, Feb 7, 2020, at 13:03, Todd wrote: approaches like opening within the dump or load, which the wiki still recommends [1].
So something like:
with open('myfile.p', 'wb') as f: pickle.dump(myobj, f)
Would be:
pickle.dump(myobj, 'myfile.p')
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))?
Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
Exactly how much code would be wrapped in the 'with' block?
I think the answer there is pretty obvious: it can only be the entire statement that the with expression is contained in. That does raise an issue for lambdas and comprehensions. I think for that case, you need to stop thinking of it as a syntactic transformation and instead think of the semantics directly: the with affects the innermost scope if it’s smaller than the innermost statement, even though there’s no way to rewrite it like that. But then nothing else in Python is defined as a syntactic rewrite, so I don’t think that’s a problem.
On Sat, Feb 8, 2020 at 1:22 PM Chris Angelico
On Sun, Feb 9, 2020 at 8:04 AM Random832
wrote: On Fri, Feb 7, 2020, at 13:03, Todd wrote:
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))?
Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
Exactly how much code would be wrapped in the 'with' block?
This is an intriguing idea, and in the example it's fairly easy to wrap the entire statement in the with block. It gets a bit more complicated with short-circuit logic. This is a contrived example to make it easier to read: result = (with alpha()) and ((with beta()) if (with gamma()) else (with delta())) needs to be interpreted something like: with _alpha := alpha(): if _alpha: with _gamma:= gamma(): if _gamma: with _beta := beta(): result = beta else: with _delta := delta(): result = delta else: result = _alpha I don't think there's anything surprising there although precisely defining the semantics will be a little tricky. --- Bruce
On 2020-02-08 6:53 p.m., Bruce Leban wrote:
On Sat, Feb 8, 2020 at 1:22 PM Chris Angelico
mailto:rosuav@gmail.com> wrote: On Sun, Feb 9, 2020 at 8:04 AM Random832
mailto:random832@fastmail.com> wrote: > > On Fri, Feb 7, 2020, at 13:03, Todd wrote: > What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))? > > Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block. >
Exactly how much code would be wrapped in the 'with' block?
This is an intriguing idea, and in the example it's fairly easy to wrap the entire statement in the with block. It gets a bit more complicated with short-circuit logic. This is a contrived example to make it easier to read:
result = (with alpha()) and ((with beta()) if (with gamma()) else (with delta()))
needs to be interpreted something like:
with _alpha := alpha():
if _alpha:
with _gamma:= gamma():
if _gamma:
with _beta := beta():
result = beta
else:
with _delta := delta():
result = delta
else:
result = _alpha
I don't think there's anything surprising there although precisely defining the semantics will be a little tricky.
I'd expect it to go more like with alpha() as _alpha: if _alpha: with gamma() as _gamma: if _gamma: with beta() as _beta: result = _beta else: with delta() as _delta: result = _delta else: result = _alpha and even then I might've messed something up here. This seems about as hard to transform as the halting problem tho, at face value.
--- Bruce
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QKG6VA... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, Feb 8, 2020, at 17:14, Soni L. wrote:
On 2020-02-08 6:53 p.m., Bruce Leban wrote:
On Sat, Feb 8, 2020 at 1:22 PM Chris Angelico
wrote: Exactly how much code would be wrapped in the 'with' block?
This is an intriguing idea, and in the example it's fairly easy to wrap the entire statement in the with block. It gets a bit more complicated with short-circuit logic. This is a contrived example to make it easier to read:
result = (with alpha()) and ((with beta()) if (with gamma()) else (with delta()))
needs to be interpreted something like: (snip) I don't think there's anything surprising there although precisely defining the semantics will be a little tricky.
I'd expect it to go more like (snip)
My own expectation, for what it's worth, would be something like try: _alpha_set = _beta_set = _gamma_set = _delta_set = False result = (_alpha_cm := alpha(), _alpha_set:=True)[0].__enter__() and ((same transform for beta) if (...gamma) else (...delta)) finally: try: if _delta_set: _delta_cm.__exit__() finally: try: if _gamma_set: _gamma_cm.__exit__() finally: ... but this is more of a mess than I originally thought to define in scenarios with multiple and/or conditionally-used context managers. It's also tempting to try to define a way to, e.g. only include it in scope for the evaluation of the condition in if statements and while loops.
On Sat, Feb 8, 2020, at 18:06, Random832 wrote:
My own expectation, for what it's worth, would be something like [snip]
After thinking about it some more, I realized that my idea can basically be translated to "all with-expressions in the statement get added to an implicit ExitStack, which is then cleaned up after the statement."
It's also tempting to try to define a way to, e.g. only include it in scope for the evaluation of the condition in if statements and while loops.
On further reflection, I don't think this is worth it in the general case - though the idea of a context manager being opened in each iteration of a while loop's condition not being closed until the end of the loop is concerning.
participants (9)
-
Andrew Barnert
-
Antoine Pitrou
-
Bruce Leban
-
Chris Angelico
-
Christopher Barker
-
Random832
-
Serhiy Storchaka
-
Soni L.
-
Todd