<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi <span style="font-family: arial, helvetica, sans-serif;" class="">Francesc</span>, </div><div class="">thanks a lot for you reply and for your impressive job on bcolz! </div><div class=""><br class=""></div><div class="">Bcolz seems to make stress on compression, which is not of much interest for me, but the <i class="">ctable</i>, and chunked operations look very appropriate to me now. (Of course, I'll need to test it much before I can say this for sure, that's current impression).</div><div class=""><br class=""></div><div class="">The strongest concern with bcolz so far is that it seems to be completely non-trivial to install on windows systems, while pip provides binaries for most (or all?) OS for numpy. </div><div class="">I didn't build pip binary wheels myself, but is it hard / impossible to cook pip-installabel binaries?</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">You can change shapes of numpy arrays, but that usually involves copies of the whole container.</div></div></div></div></div></blockquote>sure, but this is ok for me, as I plan to organize column editing in 'batches', so this should require seldom copying. </div><div class="">It would be nice to see an example to understand how deep I need to go inside numpy.</div><div class=""><br class=""></div><div class="">Cheers, </div><div class="">Alex. </div><div class=""> </div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">22 февр. 2017 г., в 17:03, Francesc Alted <<a href="mailto:faltet@gmail.com" class="">faltet@gmail.com</a>> написал(а):</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">Hi Alex,</div><div class="gmail_extra"><br class=""><div class="gmail_quote">2017-02-22 12:45 GMT+01:00 Alex Rogozhnikov<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:alex.rogozhnikov@yandex.ru" target="_blank" class="">alex.rogozhnikov@yandex.ru</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class="">Hi Nathaniel, </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div dir="auto" class=""><div dir="auto" class="">pandas</div></div></blockquote><br class=""></div><div class="">yup, the idea was to have minimal pandas.DataFrame-like storage (which I was using for a long time), </div><div class="">but without irritating problems with its row indexing and some other problems like interaction with matplotlib.</div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class=""><div dir="auto" class=""><div dir="auto" class="">A dict of arrays?</div></div></blockquote></div><div class=""><br class=""></div><div class="">that's what I've started from and implemented, but at some point I decided that I'm reinventing the wheel and numpy has something already. In principle, I can ignore this 'column-oriented' storage requirement, but potentially it may turn out to be quite slow-ish if dtype's size is large.</div></div><div class=""><br class=""></div><div class="">Suggestions are welcome.</div></div></blockquote><div class=""><br class=""></div><div class=""><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">You may want to try bcolz:</div><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;"><br class=""></div><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;"><a href="https://github.com/Blosc/bcolz" class="">https://github.com/Blosc/bcolz</a></div><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;"><br class=""></div><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">bcolz is a columnar storage, basically as you require, but data is compressed by default even when stored in-memory (although you can disable compression if you want to).</div><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">Another strange question:</div><div class="">in general, it is considered that once numpy.array is created, it's shape not changed. </div><div class="">But if i want to keep the same recarray and change it's dtype and/or shape, is there a way to do this?</div></div></blockquote><div class=""><br class=""></div><div class=""><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">You can change shapes of numpy arrays, but that usually involves copies of the whole container. With bcolz you can change length and add/del columns without copies. If your containers are large, it is better to inform bcolz on its final estimated size. See:</div><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;"><br class=""></div><div class="gmail_default"><font face="arial, helvetica, sans-serif" class=""><a href="http://bcolz.blosc.org/en/latest/opt-tips.html" class="">http://bcolz.blosc.org/en/latest/opt-tips.html</a></font><br class=""></div><br class=""></div><div class=""><div class="gmail_default" style="font-family: arial, helvetica, sans-serif;">Francesc</div></div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">Thanks, </div><div class="">Alex.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">22 февр. 2017 г., в 3:53, Nathaniel Smith <<a href="mailto:njs@pobox.com" target="_blank" class="">njs@pobox.com</a>> написал(а):</div><br class="gmail-m_2817944884559285036Apple-interchange-newline"><div class=""><div class=""><div class="gmail-h5"><div dir="auto" class=""><div class=""><div class="gmail_extra"><div class="gmail_quote">On Feb 21, 2017 3:24 PM, "Alex Rogozhnikov" <<a href="mailto:alex.rogozhnikov@yandex.ru" target="_blank" class="">alex.rogozhnikov@yandex.ru</a>> wrote:<br type="attribution" class=""><blockquote class="gmail-m_2817944884559285036quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class="">Ah, got it. Thanks, Chris!</div><div class="">I thought recarray can be only one-dimensional (like tables with named columns).</div><div class=""><br class=""></div><div class="">Maybe it's better to ask directly what I was looking for: </div><div class="">something that works like a table with named columns (but no labelling for rows), and keeps data (of different dtypes) in a column-by-column way (and this is numpy, not pandas). </div><div class=""><br class=""></div><div class="">Is there such a magic thing?</div></div></blockquote></div></div></div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Well, that's what pandas is for...</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">A dict of arrays?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">-n</div></div></div></div><span class="gmail-">______________________________<wbr class="">_________________<br class="">NumPy-Discussion mailing list<br class=""><a href="mailto:NumPy-Discussion@scipy.org" target="_blank" class="">NumPy-Discussion@scipy.org</a><br class=""><a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank" class="">https://mail.scipy.org/<wbr class="">mailman/listinfo/numpy-<wbr class="">discussion</a><br class=""></span></div></blockquote></div><br class=""></div><br class="">______________________________<wbr class="">_________________<br class="">NumPy-Discussion mailing list<br class=""><a href="mailto:NumPy-Discussion@scipy.org" class="">NumPy-Discussion@scipy.org</a><br class=""><a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank" class="">https://mail.scipy.org/<wbr class="">mailman/listinfo/numpy-<wbr class="">discussion</a><br class=""><br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature">Francesc Alted</div></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">NumPy-Discussion mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:NumPy-Discussion@scipy.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">NumPy-Discussion@scipy.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://mail.scipy.org/mailman/listinfo/numpy-discussion" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://mail.scipy.org/mailman/listinfo/numpy-discussion</a></div></blockquote></div><br class=""></body></html>