Pickle to/from filename or path
Currently with pickle you have to "open" the file separately from the pickle operation, generally using a "with" clause, then provide the file object to one of the pickle.dump or pickle.load. This adds quite a bit of boilerplate for simply saving a file I think it would be nice if pickle.dump and pickle.load would also accept a filename or path. The functions would automatically open the file in the correct way, save/load the object, then close the file. This would reduce the amount of code needed to save a single object by at least half, and would eliminate the incentive to use unsafe (I think?) approaches like opening within the dump or load, which the wiki still recommends [1]. So something like: with open('myfile.p', 'wb') as f: pickle.dump(myobj, f) Would be: pickle.dump(myobj, 'myfile.p') Sorry if this has already been discussed, but I searched the mailing list history and couldn't find it. [1] https://wiki.python.org/moin/UsingPickle
On Fri, 7 Feb 2020 13:03:52 -0500 Todd <toddrjen@gmail.com> wrote:
What you call "quite a bit of boilerplate" is just an additional line of code. If you are frequently being bothered by this (I'm curious what the context is?), it's easy to add the desired helper function to your own library. In general, I don't think adding tons of trivial convenience helpers is good API design, it ends up making the API more difficult to discover and digest. Regards Antoine.
Frankly, I’ve often wanted this for other things that share a similar API, JSON comes to mind. Loading from/saving to a file is a pretty common thing— nice to make it easy. And JSON at least has *s versions for strings, so no confusion there. But it’s been this way a LONG time, and with a general trend of Python becoming more of a systems that scripting language, I don’t see it changing. -CHB PS: you don’t need the context manager — at least in cPython — you can put the open call right in the call to load() -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Feb 7, 2020, at 10:21, Todd <toddrjen@gmail.com> wrote:
I think it would be nice if pickle.dump and pickle.load would also accept a filename or path.
Pickle intentionally has the same API as marshal, and so do a number of other modules—not just in the stdlib, but popular third party modules. So, I don’t think you’d want to change pickle to be different from everything else, but it might be a major process to get everything changed. Also, while there is code in Python that does this “file object or path” thing, I think it’s mostly older code, and not something to encourage going forward. It can be a confusing API, since there’s _other_ code that does the “file object or string” API (not too much of a problem for dump, but think about load). Maybe a keyword-only parameter would be better? Or even a third function: load for file, loads for string, loadp or load_path for path? Or maybe it would be better to have a generic wrapper that can take any file-using function and give you a path-using function: def opener(func, mode='r') @wraps(func) def wrapper(*args, **kw): with open(args[-1], mode) as f: return func(*args[:-1], f, **kwargs) This seems a little magical to have in the stdlib (especially since obviously not every function that wants a file wants it as the last positional argument—in a different language I’d probably make this take the first argument and then wrap load in opener but dump in flip opener flip…), but might be worth putting in your own toolkit.
On Fri, Feb 7, 2020, at 13:03, Todd wrote:
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))? Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
On Feb 8, 2020, at 13:25, Chris Angelico <rosuav@gmail.com> wrote:
I think the answer there is pretty obvious: it can only be the entire statement that the with expression is contained in. That does raise an issue for lambdas and comprehensions. I think for that case, you need to stop thinking of it as a syntactic transformation and instead think of the semantics directly: the with affects the innermost scope if it’s smaller than the innermost statement, even though there’s no way to rewrite it like that. But then nothing else in Python is defined as a syntactic rewrite, so I don’t think that’s a problem.
On Sat, Feb 8, 2020 at 1:22 PM Chris Angelico <rosuav@gmail.com> wrote:
This is an intriguing idea, and in the example it's fairly easy to wrap the entire statement in the with block. It gets a bit more complicated with short-circuit logic. This is a contrived example to make it easier to read: result = (with alpha()) and ((with beta()) if (with gamma()) else (with delta())) needs to be interpreted something like: with _alpha := alpha(): if _alpha: with _gamma:= gamma(): if _gamma: with _beta := beta(): result = beta else: with _delta := delta(): result = delta else: result = _alpha I don't think there's anything surprising there although precisely defining the semantics will be a little tricky. --- Bruce
On 2020-02-08 6:53 p.m., Bruce Leban wrote:
I'd expect it to go more like with alpha() as _alpha: if _alpha: with gamma() as _gamma: if _gamma: with beta() as _beta: result = _beta else: with delta() as _delta: result = _delta else: result = _alpha and even then I might've messed something up here. This seems about as hard to transform as the halting problem tho, at face value.
On Sat, Feb 8, 2020, at 17:14, Soni L. wrote:
My own expectation, for what it's worth, would be something like try: _alpha_set = _beta_set = _gamma_set = _delta_set = False result = (_alpha_cm := alpha(), _alpha_set:=True)[0].__enter__() and ((same transform for beta) if (...gamma) else (...delta)) finally: try: if _delta_set: _delta_cm.__exit__() finally: try: if _gamma_set: _gamma_cm.__exit__() finally: ... but this is more of a mess than I originally thought to define in scenarios with multiple and/or conditionally-used context managers. It's also tempting to try to define a way to, e.g. only include it in scope for the evaluation of the condition in if statements and while loops.
On Sat, Feb 8, 2020, at 18:06, Random832 wrote:
My own expectation, for what it's worth, would be something like [snip]
After thinking about it some more, I realized that my idea can basically be translated to "all with-expressions in the statement get added to an implicit ExitStack, which is then cleaned up after the statement."
On further reflection, I don't think this is worth it in the general case - though the idea of a context manager being opened in each iteration of a while loop's condition not being closed until the end of the loop is concerning.
On Fri, 7 Feb 2020 13:03:52 -0500 Todd <toddrjen@gmail.com> wrote:
What you call "quite a bit of boilerplate" is just an additional line of code. If you are frequently being bothered by this (I'm curious what the context is?), it's easy to add the desired helper function to your own library. In general, I don't think adding tons of trivial convenience helpers is good API design, it ends up making the API more difficult to discover and digest. Regards Antoine.
Frankly, I’ve often wanted this for other things that share a similar API, JSON comes to mind. Loading from/saving to a file is a pretty common thing— nice to make it easy. And JSON at least has *s versions for strings, so no confusion there. But it’s been this way a LONG time, and with a general trend of Python becoming more of a systems that scripting language, I don’t see it changing. -CHB PS: you don’t need the context manager — at least in cPython — you can put the open call right in the call to load() -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Feb 7, 2020, at 10:21, Todd <toddrjen@gmail.com> wrote:
I think it would be nice if pickle.dump and pickle.load would also accept a filename or path.
Pickle intentionally has the same API as marshal, and so do a number of other modules—not just in the stdlib, but popular third party modules. So, I don’t think you’d want to change pickle to be different from everything else, but it might be a major process to get everything changed. Also, while there is code in Python that does this “file object or path” thing, I think it’s mostly older code, and not something to encourage going forward. It can be a confusing API, since there’s _other_ code that does the “file object or string” API (not too much of a problem for dump, but think about load). Maybe a keyword-only parameter would be better? Or even a third function: load for file, loads for string, loadp or load_path for path? Or maybe it would be better to have a generic wrapper that can take any file-using function and give you a path-using function: def opener(func, mode='r') @wraps(func) def wrapper(*args, **kw): with open(args[-1], mode) as f: return func(*args[:-1], f, **kwargs) This seems a little magical to have in the stdlib (especially since obviously not every function that wants a file wants it as the last positional argument—in a different language I’d probably make this take the first argument and then wrap load in opener but dump in flip opener flip…), but might be worth putting in your own toolkit.
On Fri, Feb 7, 2020, at 13:03, Todd wrote:
What if you could write pickle.dump(myobj, with open('myfile.p', 'wb'))? Or other similar examples such as (with open('myfile')).read() - have the compiler automatically transform the code into equivalent to wrapping the statement in a with block.
On Feb 8, 2020, at 13:25, Chris Angelico <rosuav@gmail.com> wrote:
I think the answer there is pretty obvious: it can only be the entire statement that the with expression is contained in. That does raise an issue for lambdas and comprehensions. I think for that case, you need to stop thinking of it as a syntactic transformation and instead think of the semantics directly: the with affects the innermost scope if it’s smaller than the innermost statement, even though there’s no way to rewrite it like that. But then nothing else in Python is defined as a syntactic rewrite, so I don’t think that’s a problem.
On Sat, Feb 8, 2020 at 1:22 PM Chris Angelico <rosuav@gmail.com> wrote:
This is an intriguing idea, and in the example it's fairly easy to wrap the entire statement in the with block. It gets a bit more complicated with short-circuit logic. This is a contrived example to make it easier to read: result = (with alpha()) and ((with beta()) if (with gamma()) else (with delta())) needs to be interpreted something like: with _alpha := alpha(): if _alpha: with _gamma:= gamma(): if _gamma: with _beta := beta(): result = beta else: with _delta := delta(): result = delta else: result = _alpha I don't think there's anything surprising there although precisely defining the semantics will be a little tricky. --- Bruce
On 2020-02-08 6:53 p.m., Bruce Leban wrote:
I'd expect it to go more like with alpha() as _alpha: if _alpha: with gamma() as _gamma: if _gamma: with beta() as _beta: result = _beta else: with delta() as _delta: result = _delta else: result = _alpha and even then I might've messed something up here. This seems about as hard to transform as the halting problem tho, at face value.
On Sat, Feb 8, 2020, at 17:14, Soni L. wrote:
My own expectation, for what it's worth, would be something like try: _alpha_set = _beta_set = _gamma_set = _delta_set = False result = (_alpha_cm := alpha(), _alpha_set:=True)[0].__enter__() and ((same transform for beta) if (...gamma) else (...delta)) finally: try: if _delta_set: _delta_cm.__exit__() finally: try: if _gamma_set: _gamma_cm.__exit__() finally: ... but this is more of a mess than I originally thought to define in scenarios with multiple and/or conditionally-used context managers. It's also tempting to try to define a way to, e.g. only include it in scope for the evaluation of the condition in if statements and while loops.
On Sat, Feb 8, 2020, at 18:06, Random832 wrote:
My own expectation, for what it's worth, would be something like [snip]
After thinking about it some more, I realized that my idea can basically be translated to "all with-expressions in the statement get added to an implicit ExitStack, which is then cleaned up after the statement."
On further reflection, I don't think this is worth it in the general case - though the idea of a context manager being opened in each iteration of a while loop's condition not being closed until the end of the loop is concerning.
participants (9)
-
Andrew Barnert -
Antoine Pitrou -
Bruce Leban -
Chris Angelico -
Christopher Barker -
Random832 -
Serhiy Storchaka -
Soni L. -
Todd