Mailman 3 January 2021 - NumPy-Discussion

NEP 43 – How should ufunc (inner loops) be designed?
by Sebastian Berg 07 Jan '21

07 Jan '21

Hi all, I would like to nudge a discuss on NEP 43. Right now, mainly the first part: That is everything up to "Promotion and dispatching". (This is because promotion and dispatching should be the last bigger step in refactoring NumPy.) The full text is at: https://numpy.org/neps/nep-0043-extensible-ufuncs.html which includes more information and gives some background to help understand what is going on. I personally believe that the `ArrayMethod` concept is pretty much necessary to group dtype resolution with the loop functionality (and other things currently handled by the `ufunc` which may be specific to the dtypes involved, e.g. the `identity` could go here). But how exactly it should look like and especially how we want the inner-loop signature to be defined is flexible. So comments on all parts of the NEP are very welcome! My main focus would be on the `strided_inner_loop`. That is largely a continuation of this old, but pretty insightful discussion: https://github.com/numpy/numpy/issues/12518 My current proposal (C-API) is this: def strided_inner_loop( context, data, dimensions, strides, innerloop_data): return 0 # or -1 on error Where `context` is a structure holding some useful information: * The `ArrayMethod` itself (pretty much the `self`) * the dtypes/descriptors of all arrays * the original caller/ufunc (if applicable, useful for backcompat and error messages) * anything else that may come up in the future and should be passed here (e.g. the Python interpreter state). * (Possibly gufunc signature unless we duplicate it on the ArrayMethod.) Context may seem a bit weird, but passing dtypes is very convenient e.g. for strings, passing `self`, the ArrayMethod (so to speak) may also be necessary (e.g. for Python wrapping). And a struct means we could expand it in the future, with some care [1]. Admittedly, numeric ufuncs can simply ignore this completely. `data, dimensions, strides` is the main part of the existing signature. `innerloop_data` is a user allocated/defined struct that is passed in (and has its own "free" slot). We currently use this for casting in NumPy (so it does not need to be public right away, `context` should be good enough for most things). In the NEP I am considering that to pass `npy_intp *scratch_var = 0` when this is unused. That would allow persistent flags (e.g. whether a warning was already given), which otherwise would require a lot of additional complexity. How this `innerloop_data` should look like is maybe the thing I am least sold on myself. The current proposal just keeps around what is already there as private API. There are many more issues and potential future additions and I am happy to discuss those as well. I am singling out the signature a bit, because it seems particularly central and painful to change again in the future. Another example is how the signature of `get_loop()` should look like, since it would be nice to eventually make it public (or even allow overriding it) [2]. That is lower priority though, since we could get away with keeping it private API for a while. Cheers, Sebastian [1] That may need some thought: If others provide the struct (and not just consume it like a ufunc/dtype author), they would need to ensure ABI compatibility with NumPy and always "catch up" when NumPy extends the struct. I expect that can be managed, but I have not planned on how do achieve it. The thought is that e.g. sparse arrays could use the NumPy ArrayMethod/loops directly, and then they would have to provide the struct. (Those projects probably can easily catch up, but you would need to ensure a good error when a NumPy version happens to be too new and is thus incompatible.) [2] The things I am worried about there are: * How we do alignment, largest power-of-two alignment would seem nice, but that is really per-operand: https://github.com/numpy/numpy/issues/17359 * The `move_references` is probably useful flag for buffering of data that contains Python objects/references (it is used because the buffer can be cleared out immediately). For now we can say that this is only relevant for NumPy itself. But the mechanism itself does probably make sense in the context of buffered loops (i.e. casting) – I am honestly not certain that it is actually worth much in performance/memory to clear out buffers in one go.

1 0

ANN: pvlib-0.8.1 released
by Dr. Mark Alexander Mikofski PhD 06 Jan '21

06 Jan '21

Dear Pythonistas and solar power enthusiasts, On behalf of the maintainers, we're happy to announce a new release of pvlib python: software for simulating performance of photovoltaic solar energy systems. *See what's new for v0.8.1:* * https://pvlib-python.readthedocs.io/en/stable/whatsnew.html *Releases are available from PyPI and the conda-forge channel:* * https://pypi.org/project/pvlib/ * https://anaconda.org/conda-forge/pvlib-python *Read the Documentation:* * https://pvlib-python.readthedocs.io/en/stable/index.html *Report issues & contribute:* * https://github.com/pvlib/pvlib-python *Highlights:* * AirSpeed Velocity benchmarks are now available! https://pvlib-benchmarker.github.io/pvlib-benchmarks/ * A numpy-based implementation of the SPECTRL2 spectral irradiance model and a usage example in the gallery. https://pvlib-python.readthedocs.io/en/stable/auto_examples/plot_spectrl2_f… * New functions for modeling inverters with multiple MPPTs. * The Liu-Jordan irradiance function is deprecated, and will be removed in v0.9. Please use the newer Campbell-Norman function instead, but note it expects some different arguments. * Added the option of fetching 5 and 15-minute PSM3 data using a new "attributes" keyword argument. * Clear-sky detection now uses centered rolling windows instead of left-aligned rolling windows. *The maintainers thank you for using pvlib python!* -- Mark Mikofski, PhD (2005) *Fiat Lux*

1 0

NumPy Community Meeting Wednesday
by Sebastian Berg 05 Jan '21

05 Jan '21

Hi all, There will be a NumPy Community meeting Wednesday January 6th at 12pm Pacific Time (20:00 UTC). Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes at: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

NumPy 1.19.5 released
by Charles R Harris 05 Jan '21

05 Jan '21

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.19.5. NumPy 1.19.5 is a short bugfix release. Apart from fixing several bugs, the main improvement is an update to OpenBLAS 0.3.13 that works around the Windows 2004 fmod bug while not breaking execution on other platforms. This release supports Python 3.6-3.9 and is planned to be the last release in the 1.19.x cycle. NumPy Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.19.5/>, source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.19.5>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Christoph Gohlke - Matti Picus - Raghuveer Devulapalli - Sebastian Berg - Simon Graham + - Veniamin Petrenko + - Bernie Gray + *Pull requests merged* A total of 11 pull requests were merged for this release. - #17756: BUG: Fix segfault due to out of bound pointer in floatstatus... - #17774: BUG: fix np.timedelta64('nat').__format__ throwing an exception - #17775: BUG: Fixed file handle leak in array_tofile. - #17786: BUG: Raise recursion error during dimension discovery - #17917: BUG: Fix subarray dtype used with too large count in fromfile - #17918: BUG: 'bool' object has no attribute 'ndim' - #17919: BUG: ensure _UFuncNoLoopError can be pickled - #17924: BLD: use BUFFERSIZE=20 in OpenBLAS - #18026: BLD: update to OpenBLAS 0.3.13 - #18036: BUG: make a variable volatile to work around clang compiler bug - #18114: REL: Prepare for the NumPy 1.19.5 release. Cheers, Charles Harris

1 0

Speedups for intersect1d in numpy.lib
by Stamm, Felix 04 Jan '21

04 Jan '21

Hello mailing list, I would like to open a PR that improves the speed of intersect1d in particular for large arrays of potentially unequal sizes. As parts of the change would involve a new keyword I think I should write to this mailing list first (correct me if I am wrong). Lets start with the current implementation of intersect1d. The steps are (roughly): I) concatenate both arrays II) sort that large array and III) keep only duplicates. This is a simple but not necessary fast implementation. In particular it has has complexity O( (len1 + len2) log(len1 + len2)). I have two suggestions one of which can greatly improve the runtime (1) while the other has some minor improvements (2). Lets start with the major one. 1) Adding assume_sorted as keyword argument. If you can already assume that the arrays are sorted you can improve the runtime to min(len1, len2)*log(max(len1, len2)) which is orders of magnitude faster especially when the arrays intersected are unequal in size. The implementation basically is I) figure out which is the smaller array II) make that array unique III) use np.searchsorted to find duplicates of the smaller in the larger. I think that having assume_sorted as a keyword argument is quite useful, especially as the result of assume_sorted is always sorted. So chaining multiple intersections together can be made much faster. 2) Improve the performance in the case that we cannot assume sorted arrays In the case that we cannot assume that both arrays are already sorted we can still gain advantages. Mainly by sorting the arrays separately and then using np.searchsorted again. This brings the complexity down to O( len1 log(len1) + len2 log(len2)) + min(len1, len2)*log(max(len1, len2)). I already mention this in the issue 16964: https://github.com/numpy/numpy/issues/16964 Looking forward for your feedback. Have a great day and stay safe Felix Stamm

1 0

proposal for NumPy sponsorship guidelines (NEP 46)
by Ralf Gommers 03 Jan '21

03 Jan '21

Hi all, I just opened https://github.com/numpy/numpy/pull/18084, "NumPy Sponsorship Guidelines". Below are the most important parts for review (for Related Work, References, etc. see the PR). Please bring up broader points here, and small/textual feedback on the PR. Cheers, Ralf Abstract -------- This NEP provides guidelines on how the NumPy project will acknowledge financial and in-kind support. Motivation and Scope -------------------- In the past few years the NumPy project has gotten significant financial support, as well as dedicated work time for maintainers to work on NumPy. There is a need to acknowledge that support - funders and organizations expect or require it, it's helpful when looking for new funding, and it's the right thing to do. Furthermore, having a clear policy for how NumPy acknowledges support is helpful when searching for new support. This NEP is aimed at both the NumPy community - who can use it when looking for support and acknowledging existing support - and at past, current and prospective sponsors, who often want or need to know what they get in return for their support (other than a healthier NumPy). The scope of this proposal includes: - direct financial support, employers providing paid time for NumPy maintainers and regular contributors, and in-kind support such as free hardware resources or services. - where and how NumPy acknowledges support (e.g., logo placement on the website). - the amount and duration of support which leads to acknowledgement. - who in the NumPy project is responsible for sponsorship related topics, and how to contact them. How NumPy will acknowledge support ---------------------------------- There will be two different ways to acknowledge financial and in-kind support, one to recognize significant active support, and another one to recognize support received in the past and smaller amounts of support. Entities who fall under "significant active supporter" we'll call Sponsor. The minimum level of support given to NumPy to be considered a Sponsor are: - $30,000/yr for unrestricted financial contributions - $60,000/yr for financial contributions for a particular purpose - $100,000/yr for in-kind contributions The rationale for the above levels is that unrestricted financial contributions are typically the most valuable for the project, and the hardest to obtain. The opposite is true for in-kind contributions. The dollar value of the levels also reflect that NumPy's needs have grown to the point where we need at least a few paid developers in order to effectively support our user base and continue to move the project forward. Financial support at or above these levels is needed to be able to make a significant difference. Sponsors will get acknowledged through: - a small logo displayed on the front page of the NumPy website - prominent logo placement on https://numpy.org/about/ - logos displayed in talks about NumPy by maintainers - announcements of the sponsorship on the NumPy mailing list and the numpy-team Twitter account In addition to Sponsors, we already have the concept of Institutional Partner (defined in NumPy's `governance document <https://numpy.org/devdocs/dev/governance/index.html >`__), for entities who employ a NumPy maintainer and let them work on NumPy as part of their official duties. The governance document doesn't currently define a minimum amount of paid maintainer time needed to be considered for partnership. Therefore we propose that level here, roughly in line with the sponsorship levels: - 6 person-months/yr of paid work time for one or more NumPy maintainers or regular contributors Institutional Partners get the same benefits as Sponsors, in addition to what is specified in the NumPy governance document. Finally, a new page on the website (https://numpy.org/funding/, linked from the About page) will be added to acknowledge all current and previous sponsors, partners, and any other entities and individuals who provided $5,000 or more of financial or in-kind support. This page will include relevant details of support (dates, amounts, names and purpose); no logos will be used on this page. The rationale for the $5,000 minimum level is to keep the amount of work maintaining the page reasonable; the level is the equivalent of, e.g., one GSoC or a person-week's worth of engineering time in a Western country, which seems like a reasonable lower limit. Implementation -------------- The following content changes need to be made: - Add a section with small logos towards the bottom of the `numpy.org <https://numpy.org/>`__ website. - Create a full list of historical and current support and deploy it to https://numpy.org/funding. - Update the NumPy governance document for changes to Institutional Partner eligibility requirements and benefits. - Update https://numpy.org/about with details on how to get in touch with the NumPy project about sponsorship related matters (see next section). A NumPy Funding Team ~~~~~~~~~~~~~~~~~~~~ At the moment NumPy has only one official body, the Steering Council, and no good way to get in touch with either that body or any person or group responsible for funding and sponsorship related matters. The way this is typically done now is to somehow find the personal email of a maintainer, and email them in private. There is a need to organize this more transparently - a potential sponsor isn't likely to inquire through the mailing list, nor is it easy for a potential sponsor to know if they're reaching out to the right person in private. https://numpy.org/about/ already says that NumPy has a "funding and grants" team, however that is not the case. We propose to organize this team, name team members on it, and add the names of those team members plus a dedicated email address for the team to the About page. Status before this proposal --------------------------- Acknowledgement of support ~~~~~~~~~~~~~~~~~~~~~~~~~~ At the time of writing (Dec 2020), the logos of the four largest financial sponsors and two institutional partners are displayed on https://numpy.org/about/. The `Nature paper about NumPy < https://www.nature.com/articles/s41586-020-2649-2>`__ mentions some early funding. No comprehensive list of received funding and in-kind support is published anywhere. Decisions on which logos to list on the website have been made mostly by the website team. Decisions on which entities to recognize as Institutional Partner have been made by the NumPy Steering Council. NumPy governance, decision-making and financial oversight ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *This section is meant as context for the reader, to help put the rest of this NEP in perspective, and perhaps answer questions the reader has when reading this as a potential sponsor.* NumPy has a formal governance structure defined in `this governance document < https://numpy.org/devdocs/dev/governance/index.html>`__). Decisions are made by consensus among all active participants in a discussion (typically on the mailing list), and if consensus cannot be reached then the Steering Council takes the decision (also by consensus). NumPy is a sponsored project of NumFOCUS, a US-based 501(c)3 nonprofit. NumFOCUS administers NumPy funds, and ensures they are spent in accordance with its mission and nonprofit status. In practice, NumPy has a NumFOCUS subcommittee (with its members named in the NumPy governance document) who can authorize financial transactions. Those transactions, for example paying a contractor for a particular activity or deliverable, are decided on by the NumPy Steering Council. Alternatives ------------ *Tiered sponsorship levels.* We considered using tiered sponsorship levels, and rejected this alternative because it would be more complex, and not necessarily communicate the right intent - the minimum levels are for us to determine how to acknowledge support that we receive, not a commercial value proposition. Entities typically will support NumPy because they rely on the project or want to help advance it, and not to get brand awareness through logo placement. *Listing all donations*. Note that in the past we have received many smaller donations, mostly from individuals through NumFOCUS. It would be great to list all of those contributions, but given the way we receive information on those donations right now, that would be quite labor-intensive. If we manage to move to a more suitable platform, such as `Open Collective < https://opencollective.com/>`__, in the future, we should reconsider listing all individual donations.

2 2