Sorry, typo: I can send a PR or *review* one, if that helps. ср, 15 июл. 2020 г., 11:29 Evgeni Burovski <evgeny.burovskiy@gmail.com>:
While it's not wholly surprising these two are slow, it is surprising they are *that* slow.
W.r.t. docstrings, I think there's room for adding a "skip_focstring" kwarg or some such to rv_generic. It'll need to be propagated to `rv_frozen.dist`. I can send a PR or one, if that helps. (I don't know about pickling, sadly.)
All that said, maybe it's easier to share the shapes between processes and use regular distributions if rv_frozen is a bottleneck, will that help?
ср, 15 июл. 2020 г., 2:03 Andrew Nelson <andyfaff@gmail.com>:
I have some code that uses multiprocessing.Pool for parallelisation. This requires that an object is pickled. This object has an `rv_frozen` distribution as an attribute. It turns out that a performance is much improved if the `rv_frozen` distribution is not present --> pickling of `rv_frozen` objects is expensive. Creation of `rv_frozen` objects is also expensive.
```
import scipy.stats as stats import pickle %timeit stats.norm(scale=1, loc=1) 694 µs ± 123 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) rv = stats.norm(scale=1, loc=1) %timeit s = pickle.dumps(rv); pickle.loads(s) 1.02 ms ± 24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I'd be hoping for an order of magnitude less in time for either of those. Using line profiling two of the big culprits for slowness during object creation are `rv_continuous._construct_doc` (50% of the total time, with a large part spent in `_lib.doccer.docformat`!!) and `rv_continuous._construct_argparser` My questions are: 1) Is it possible to speed up pickling/unpickling of these objects? (e.g. __setstate__/__getstate__, custom reduction, copyreg magic, ...) 2) Is there any way to turn off docstring creation (or speeding it up), besides starting the interpreter with -OO? _____________________________________ Dr. Andrew Nelson _____________________________________ _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev