pickle, modules, and ImportErrors
john_ladasky at sbcglobal.net
Thu Jan 8 01:23:43 CET 2015
On Wednesday, January 7, 2015 12:56:29 PM UTC-8, Devin Jeanpierre wrote:
> If you never run model directly, and only ever import it or run it as
> my_svr.model, then you will be fine, and pickles will all serialize
> and deserialize the same way.
Thank you Devin... I re-ran TrainingSession from within ipython, and everything worked the way I hoped. I obtained an SVRModel object which I could pickle to disk, and unpickle in a subsequent session. I don't actually need to run the test code that I appended to training.py any more, so I won't.
> P.S. don't use pickle, it is a security vulnerability equivalent in
> severity to using exec in your code, and an unversioned opaque
> schemaless blob that is very difficult to work with when circumstances
For all of its shortcomings, I can't live without pickle. In this case, I am doing data mining. My TrainingSession class commandeers seven CPU cores via Multiprocessing.Pool. Still, even my "toy" TrainingSessions take several minutes to run. I can't afford to re-run TrainingSession every time I need my models. I need a persistent object.
Besides, the opportunity for mischief is low. My code is for my own personal use. And I trust the third-party libraries that I am using. My SVRModel object wraps the NuSVR object from scikit-learn, which in turn wraps the libsvm binary.
> > try:
> > from model import *
> > from sampling import *
> > except ImportError:
> > from .model import *
> > from .sampling import *
> > This bothers me. I don't know whether it is correct usage. I don't know whether it is causing my remaining ImportError problem.
> This is a symptom of the differing ways you are importing these
> modules, as above. If you only ever run them and import them as
> my_svr.blahblah, then only the second set of imports are necessary.
OK, I will try refactoring the import code at some point.
> I hope that resolves all your questions!
I think so, thanks.
More information about the Python-list