<html><head></head><body><div style="font-family:bookman old style, new york, times, serif;font-size:16px;"><div>How much of the DATASETS issues could be handled simply by references in the documentation to where users can find those datasets that are generally considered both "standard" and potentially useful, without "physically" incorporating those datasets into SciPy?</div><div><br></div><div>E.g, could the ECG dataset be handled that way?<br></div><div><br></div><div class="ydpd5104585signature">"You won't find the right answers if you don't ask the right questions!" (Robert Helmbold, 2013)</div>
<div><br></div><div><br></div>
<div class="yahoo_quoted" id="yahoo_quoted_5847718259">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Saturday, April 28, 2018 11:42:46 PM MST, scipy-dev-request@python.org <scipy-dev-request@python.org> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Send SciPy-Dev mailing list submissions to<br></div><div dir="ltr"> <a href="mailto:scipy-dev@python.org" ymailto="mailto:scipy-dev@python.org">scipy-dev@python.org</a><br></div><div dir="ltr"><br></div><div dir="ltr">To subscribe or unsubscribe via the World Wide Web, visit<br></div><div dir="ltr"> <a href="https://mail.python.org/mailman/listinfo/scipy-dev" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br></div><div dir="ltr">or, via email, send a message with subject or body 'help' to<br></div><div dir="ltr"> <a href="mailto:scipy-dev-request@python.org" ymailto="mailto:scipy-dev-request@python.org">scipy-dev-request@python.org</a><br></div><div dir="ltr"><br></div><div dir="ltr">You can reach the person managing the list at<br></div><div dir="ltr"> <a href="mailto:scipy-dev-owner@python.org" ymailto="mailto:scipy-dev-owner@python.org">scipy-dev-owner@python.org</a><br></div><div dir="ltr"><br></div><div dir="ltr">When replying, please edit your Subject line so it is more specific<br></div><div dir="ltr">than "Re: Contents of SciPy-Dev digest..."<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Today's Topics:<br></div><div dir="ltr"><br></div><div dir="ltr"> 1. Re: New subpackage: scipy.data (Ralf Gommers)<br></div><div dir="ltr"> 2. Re: New subpackage: scipy.data (Robert Kern)<br></div><div dir="ltr"> 3. Re: New subpackage: scipy.data (Ralf Gommers)<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">----------------------------------------------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 1<br></div><div dir="ltr">Date: Sat, 28 Apr 2018 22:58:44 -0700<br></div><div dir="ltr">From: Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com" ymailto="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>><br></div><div dir="ltr">To: SciPy Developers List <<a href="mailto:scipy-dev@python.org" ymailto="mailto:scipy-dev@python.org">scipy-dev@python.org</a>><br></div><div dir="ltr">Subject: Re: [SciPy-Dev] New subpackage: scipy.data<br></div><div dir="ltr">Message-ID:<br></div><div dir="ltr"> <CABL7CQjuAKrHbVwSEWXd_V1uzLV-=XbokG=<a href="mailto:ZokMDt354hTpszw@mail.gmail.com" ymailto="mailto:ZokMDt354hTpszw@mail.gmail.com">ZokMDt354hTpszw@mail.gmail.com</a>><br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">On Tue, Apr 3, 2018 at 1:06 AM, Da?id <<a href="mailto:davidmenhur@gmail.com" ymailto="mailto:davidmenhur@gmail.com">davidmenhur@gmail.com</a>> wrote:<br></div><div dir="ltr"><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> On 31 March 2018 at 02:17, Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com" ymailto="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">>><br></div><div dir="ltr">>><br></div><div dir="ltr">>> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson <<a href="mailto:larson.eric.d@gmail.com" ymailto="mailto:larson.eric.d@gmail.com">larson.eric.d@gmail.com</a>><br></div><div dir="ltr">>> wrote:<br></div><div dir="ltr">>><br></div><div dir="ltr">>>> Top-level module for them alone sounds overkill, and I'm not sure if<br></div><div dir="ltr">>>>> discoverability alone is enough.<br></div><div dir="ltr">>>>><br></div><div dir="ltr">>>><br></div><div dir="ltr">>>> Fine by me. And if we follow the idea that these should be added<br></div><div dir="ltr">>>> sparingly, we can maintain discoverability without it growing out of<br></div><div dir="ltr">>>> hand by populating the See Also sections of each function.<br></div><div dir="ltr">>>><br></div><div dir="ltr">>><br></div><div dir="ltr">>> I agree with this, the 2 images and 1 ECG signal (to be added) that we<br></div><div dir="ltr">>> have doesn't justify a top-level module. We don't want to grow more than<br></div><div dir="ltr">>> the absolute minimum of datasets. The package is already very large, which<br></div><div dir="ltr">>> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS<br></div><div dir="ltr">>> Lambda limit of 50 MB, but there's not much margin.<br></div><div dir="ltr">>><br></div><div dir="ltr">><br></div><div dir="ltr">> The biggest subpackage is sparse, and there most of the space is taken by _<br></div><div dir="ltr">> sparsetools.cpython-35m-x86_64-linux-gnu.so According to size -A -d, the<br></div><div dir="ltr">> biggest sections are debug. The same goes for the second biggest, special.<br></div><div dir="ltr">> Can it run without those sections? On preliminary checks, it seems that<br></div><div dir="ltr">> stripping .debug_info and .debug_loc trim down the size from 38 to 3.7 MB,<br></div><div dir="ltr">> and the test suite still passes.<br></div><div dir="ltr">><br></div><div dir="ltr"><br></div><div dir="ltr">Should work. That's a lot more gain than I'd realized. Given that we hardly<br></div><div dir="ltr">ever get useful gdb tracebacks, it may be worth considering doing that for<br></div><div dir="ltr">releases.<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">><br></div><div dir="ltr">> If we really need to trim down the size for installing in things like<br></div><div dir="ltr">> Lambda, could we have a scipy-lite for production environments, that is the<br></div><div dir="ltr">> same as scipy but without unnecessary debug? I imagine tracebacks would not<br></div><div dir="ltr">> be as informative, but that shouldn't matter for production environments.<br></div><div dir="ltr">> My first thought was to remove docstrings, comments, tests, and data, but<br></div><div dir="ltr">> maybe they don't amount to so much for the trouble.<br></div><div dir="ltr">><br></div><div dir="ltr"><br></div><div dir="ltr">Recipes for such things are floating around, and it makes sense to do that.<br></div><div dir="ltr">I'd rather not maintain an official scipy-lite package though, rather just<br></div><div dir="ltr">make choices within scipy that enable third parties to do that.<br></div><div dir="ltr"><br></div><div dir="ltr">Ralf<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> On the topic at hand, I would agree to having a few, small datasets to<br></div><div dir="ltr">> showcase functionality. I think a few kilobytes can go a long way to show<br></div><div dir="ltr">> and benchmark. As far as I can see, a top level module is free: it wouldn't<br></div><div dir="ltr">> add any maintenance burden, and would make them easier to find.<br></div><div dir="ltr">><br></div><div dir="ltr">> /David.<br></div><div dir="ltr">><br></div><div dir="ltr">> _______________________________________________<br></div><div dir="ltr">> SciPy-Dev mailing list<br></div><div dir="ltr">> <a href="mailto:SciPy-Dev@python.org" ymailto="mailto:SciPy-Dev@python.org">SciPy-Dev@python.org</a><br></div><div dir="ltr">> <a href="https://mail.python.org/mailman/listinfo/scipy-dev" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">-------------- next part --------------<br></div><div dir="ltr">An HTML attachment was scrubbed...<br></div><div dir="ltr">URL: <<a href="http://mail.python.org/pipermail/scipy-dev/attachments/20180428/f1455f5b/attachment-0001.html" target="_blank">http://mail.python.org/pipermail/scipy-dev/attachments/20180428/f1455f5b/attachment-0001.html</a>><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 2<br></div><div dir="ltr">Date: Sun, 29 Apr 2018 06:21:55 +0000<br></div><div dir="ltr">From: Robert Kern <<a href="mailto:robert.kern@gmail.com" ymailto="mailto:robert.kern@gmail.com">robert.kern@gmail.com</a>><br></div><div dir="ltr">To: SciPy Developers List <<a href="mailto:scipy-dev@python.org" ymailto="mailto:scipy-dev@python.org">scipy-dev@python.org</a>><br></div><div dir="ltr">Subject: Re: [SciPy-Dev] New subpackage: scipy.data<br></div><div dir="ltr">Message-ID:<br></div><div dir="ltr"> <CAF6FJitOZ11k+epRJ8kDS7X40RDTFKou65aCGxhfTE_p=<a href="mailto:tQTyA@mail.gmail.com" ymailto="mailto:tQTyA@mail.gmail.com">tQTyA@mail.gmail.com</a>><br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com" ymailto="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>><br></div><div dir="ltr">wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">> On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser <<br></div><div dir="ltr"><a href="mailto:warren.weckesser@gmail.com" ymailto="mailto:warren.weckesser@gmail.com">warren.weckesser@gmail.com</a>> wrote:<br></div><div dir="ltr"><br></div><div dir="ltr">>> (c) We actually *use* the dataset in one of *our* docstrings or<br></div><div dir="ltr">tutorials. I don't think our datasets package should become a repository<br></div><div dir="ltr">of interesting scientific data with no connection to the scipy code. Its<br></div><div dir="ltr">purpose should be to enrich our documentation. (Note that by this<br></div><div dir="ltr">criterion, the recently added ECG signal would not qualify!)<br></div><div dir="ltr">><br></div><div dir="ltr">> I'd add the criterion that we should *only* use any dataset in the docs.<br></div><div dir="ltr">Hence there are zero internal imports, and the whole datasets submodule can<br></div><div dir="ltr">then very simply be stripped for space-constrained usage scenarios. (in<br></div><div dir="ltr">those cases a separate package would help even)<br></div><div dir="ltr"><br></div><div dir="ltr">I believe that one of the motivations for adding the ECG dataset was to<br></div><div dir="ltr">make some of the scipy.signal unit tests more realistic. Is that something<br></div><div dir="ltr">you'd like to forbid? On the one hand, if you're strapped for space, you<br></div><div dir="ltr">probably want to remove the test suites as well. On the other hand, you do<br></div><div dir="ltr">want to be able to test your stripped installation!<br></div><div dir="ltr"><br></div><div dir="ltr">--<br></div><div dir="ltr">Robert Kern<br></div><div dir="ltr">-------------- next part --------------<br></div><div dir="ltr">An HTML attachment was scrubbed...<br></div><div dir="ltr">URL: <<a href="http://mail.python.org/pipermail/scipy-dev/attachments/20180429/4adb589f/attachment-0001.html" target="_blank">http://mail.python.org/pipermail/scipy-dev/attachments/20180429/4adb589f/attachment-0001.html</a>><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 3<br></div><div dir="ltr">Date: Sat, 28 Apr 2018 23:41:39 -0700<br></div><div dir="ltr">From: Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com" ymailto="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>><br></div><div dir="ltr">To: SciPy Developers List <<a href="mailto:scipy-dev@python.org" ymailto="mailto:scipy-dev@python.org">scipy-dev@python.org</a>><br></div><div dir="ltr">Subject: Re: [SciPy-Dev] New subpackage: scipy.data<br></div><div dir="ltr">Message-ID:<br></div><div dir="ltr"> <CABL7CQgeuKMC2-o7LtJShvO2-EvV+<a href="mailto:3reKS1Tu_9N8GmRO0CvCA@mail.gmail.com" ymailto="mailto:3reKS1Tu_9N8GmRO0CvCA@mail.gmail.com">3reKS1Tu_9N8GmRO0CvCA@mail.gmail.com</a>><br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">On Sat, Apr 28, 2018 at 11:21 PM, Robert Kern <<a href="mailto:robert.kern@gmail.com" ymailto="mailto:robert.kern@gmail.com">robert.kern@gmail.com</a>> wrote:<br></div><div dir="ltr"><br></div><div dir="ltr">> On Sat, Apr 28, 2018 at 10:46 PM Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com" ymailto="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>><br></div><div dir="ltr">> wrote:<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > On Mon, Apr 2, 2018 at 11:50 AM, Warren Weckesser <<br></div><div dir="ltr">> <a href="mailto:warren.weckesser@gmail.com" ymailto="mailto:warren.weckesser@gmail.com">warren.weckesser@gmail.com</a>> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">> >> (c) We actually *use* the dataset in one of *our* docstrings or<br></div><div dir="ltr">> tutorials. I don't think our datasets package should become a repository<br></div><div dir="ltr">> of interesting scientific data with no connection to the scipy code. Its<br></div><div dir="ltr">> purpose should be to enrich our documentation. (Note that by this<br></div><div dir="ltr">> criterion, the recently added ECG signal would not qualify!)<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > I'd add the criterion that we should *only* use any dataset in the docs.<br></div><div dir="ltr">> Hence there are zero internal imports, and the whole datasets submodule can<br></div><div dir="ltr">> then very simply be stripped for space-constrained usage scenarios. (in<br></div><div dir="ltr">> those cases a separate package would help even)<br></div><div dir="ltr">><br></div><div dir="ltr">> I believe that one of the motivations for adding the ECG dataset was to<br></div><div dir="ltr">> make some of the scipy.signal unit tests more realistic. Is that something<br></div><div dir="ltr">> you'd like to forbid? On the one hand, if you're strapped for space, you<br></div><div dir="ltr">> probably want to remove the test suites as well. On the other hand, you do<br></div><div dir="ltr">> want to be able to test your stripped installation!<br></div><div dir="ltr">><br></div><div dir="ltr"><br></div><div dir="ltr">Hmm, tough question. Ideally I'd like to say yes, however we do need test<br></div><div dir="ltr">data in some cases. In practice I think one would want to strip the test<br></div><div dir="ltr">suite anyway; scipy/special/tests/data/*.npz is over 1 MB already. So let's<br></div><div dir="ltr">say that importing from within tests is okay.<br></div><div dir="ltr"><br></div><div dir="ltr">Ralf<br></div><div dir="ltr">-------------- next part --------------<br></div><div dir="ltr">An HTML attachment was scrubbed...<br></div><div dir="ltr">URL: <<a href="http://mail.python.org/pipermail/scipy-dev/attachments/20180428/bafa35fb/attachment.html" target="_blank">http://mail.python.org/pipermail/scipy-dev/attachments/20180428/bafa35fb/attachment.html</a>><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Subject: Digest Footer<br></div><div dir="ltr"><br></div><div dir="ltr">_______________________________________________<br></div><div dir="ltr">SciPy-Dev mailing list<br></div><div dir="ltr"><a href="mailto:SciPy-Dev@python.org" ymailto="mailto:SciPy-Dev@python.org">SciPy-Dev@python.org</a><br></div><div dir="ltr"><a href="https://mail.python.org/mailman/listinfo/scipy-dev" target="_blank">https://mail.python.org/mailman/listinfo/scipy-dev</a><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">End of SciPy-Dev Digest, Vol 174, Issue 31<br></div><div dir="ltr">******************************************<br></div></div>
</div>
</div></div></body></html>