Mailman 3 June 2019 - NumPy-Discussion

[JOB] Full-time opportunity – Software engineer for open source project
by Umberto Lupo 24 Jun '19

24 Jun '19

Who we are L2F is a start-up based on the EPFL Innovation Park (Lausanne, Switzerland). We are currently working at the frontier of machine learning and topological data analysis, in collaboration with several academic partners. Our Mission We are developing an open source Python library implementing innovative topological data analysis algorithms which are being designed by our team of full-time research scientists and post-doctoral researchers. The library shall be user-friendly, well documented, high-performance and well integrated with state-of-the-art machine learning libraries (such as NumPy/SciPy, scikit-learn and Keras or other popular deep learning frameworks). We are offering a full-time job in our company to help us develop this library. The candidate will work in the L2F research team. Profile description We are looking for a computer scientist matching these characteristics: * 2+ years of experience in software engineering. * Skilled with Python and C++ (in particular, at ease wrapping C++ code for Python). * Aware of how open source communities work. Better if he/she contributed in open-source collaborations, such as scikit-learn. * At ease writing specifications, developer documentation and good user documentation. * Fluent with continuous integration, Git and common developer tools. * Skilled in testing architectures (unit tests, integration tests, etc.). How to apply Applicants can write an e-mail to Dr. Matteo Caorsi (m.caorsi(a)l2f.ch<mailto:m.caorsi@l2f.ch>) attaching their CV and a short letter detailing their relevant experience and motivation. Starting date This position is available for immediate start for the right candidate.

1 0

Better way to create multiple independent random number generators
by Sebastian Berg 22 Jun '19

22 Jun '19

Hi all, since this is going to be a new addition as part of the randomgen, I thought I would just mention it on the mailing list. The Pull Request: https://github.com/numpy/numpy/pull/13780 Implements a new SeedSequence object based on Robert Kern's proposal and especially the work by Prof. O'Neill's which is included in C++. This new API allows to create many independent random generators/random number streams. For now this will be exposed by a new object: ``` entropy = {None, int, sequence[int]} # None or "seed" seed_seq = np.random.SeedSequence(entropy=entropy) # Run 100 predictable independend streams: for spawned_seed_seq in seed_seq.spawn(100): run_parallel_task(spawned_seed_seq) # where `run_parallel_task` will do: def run_parallel_task(seed_seq): # Create a BitGenerator and a Generator [1] bit_rng = np.random.PCG64(seed_seq) rng = np.random.Generator(bit_rng) ``` The beauty is that `run_parellel_task` can again use `seed_seq.spawn()` to create another set of independent streams. I am very happy with this new API. Right now we decided to opt for a SeedSequence object. In the future we may opt to adding a `.spawn()` method directly to the Generator or BitGenerator. This is mostly a heads up, since it is a new set of API, which I believe has never been mentioned/proposed on the mailing list. Best, Sebastian [1] The new API separates the BitGenerator which creates the random streams from the Generator which uses the random stream to give sample distributions of random numbers providing `uniform`, or `normal`, etc.

2 1

Moving forward with value based casting
by Sebastian Berg 21 Jun '19

21 Jun '19

Hi all, TL;DR: Value based promotion seems complex both for users and ufunc- dispatching/promotion logic. Is there any way we can move forward here, and if we do, could we just risk some possible (maybe not-existing) corner cases to break early to get on the way? ----------- Currently when you write code such as: arr = np.array([1, 43, 23], dtype=np.uint16) res = arr + 1 Numpy uses fairly sophisticated logic to decide that `1` can be represented as a uint16, and thus for all unary functions (and most others as well), the output will have a `res.dtype` of uint16. Similar logic also exists for floating point types, where a lower precision floating point can be used: arr = np.array([1, 43, 23], dtype=np.float32) (arr + np.float64(2.)).dtype # will be float32 Currently, this value based logic is enforced by checking whether the cast is possible: "4" can be cast to int8, uint8. So the first call above will at some point check if "uint16 + uint16 -> uint16" is a valid operation, find that it is, and thus stop searching. (There is the additional logic, that when both/all operands are scalars, it is not applied). Note that while it is defined in terms of casting "1" to uint8 safely being possible even though 1 may be typed as int64. This logic thus affects all promotion rules as well (i.e. what should the output dtype be). There 2 main discussion points/issues about it: 1. Should value based casting/promotion logic exist at all? Arguably an `np.int32(3)` has type information attached to it, so why should we ignore it. It can also be tricky for users, because a small change in values can change the result data type. Because 0-D arrays and scalars are too close inside numpy (you will often not know which one you get). There is not much option but to handle them identically. However, it seems pretty odd that: * `np.array(3, dtype=np.int32)` + np.arange(10, dtype=int8) * `np.array([3], dtype=np.int32)` + np.arange(10, dtype=int8) give a different result. This is a bit different for python scalars, which do not have a type attached already. 2. Promotion and type resolution in Ufuncs: What is currently bothering me is that the decision what the output dtypes should be currently depends on the values in complicated ways. It would be nice if we can decide which type signature to use without actually looking at values (or at least only very early on). One reason here is caching and simplicity. I would like to be able to cache which loop should be used for what input. Having value based casting in there bloats up the problem. Of course it currently works OK, but especially when user dtypes come into play, caching would seem like a nice optimization option. Because `uint8(127)` can also be a `int8`, but uint8(128) it is not as simple as finding the "minimal" dtype once and working with that." Of course Eric and I discussed this a bit before, and you could create an internal "uint7" dtype which has the only purpose of flagging that a cast to int8 is safe. I suppose it is possible I am barking up the wrong tree here, and this caching/predictability is not vital (or can be solved with such an internal dtype easily, although I am not sure it seems elegant). Possible options to move forward -------------------------------- I have to still see a bit how trick things are. But there are a few possible options. I would like to move the scalar logic to the beginning of ufunc calls: * The uint7 idea would be one solution * Simply implement something that works for numpy and all except strange external ufuncs (I can only think of numba as a plausible candidate for creating such). My current plan is to see where the second thing leaves me. We also should see if we cannot move the whole thing forward, in which case the main decision would have to be forward to where. My opinion is currently that when a type has a dtype associated with it clearly, we should always use that dtype in the future. This mostly means that numpy dtypes such as `np.int64` will always be treated like an int64, and never like a `uint8` because they happen to be castable to that. For values without a dtype attached (read python integers, floats), I see three options, from more complex to simpler: 1. Keep the current logic in place as much as possible 2. Only support value based promotion for operators, e.g.: `arr + scalar` may do it, but `np.add(arr, scalar)` will not. The upside is that it limits the complexity to a much simpler problem, the downside is that the ufunc call and operator match less clearly. 3. Just associate python float with float64 and python integers with long/int64 and force users to always type them explicitly if they need to. The downside of 1. is that it doesn't help with simplifying the current situation all that much, because we still have the special casting around... I have realized that this got much too long, so I hope it makes sense. I will continue to dabble along on these things a bit, so if nothing else maybe writing it helps me to get a bit clearer on things... Best, Sebastian

9 23

NumPy Community Meeting June 19
by Sebastian Berg 18 Jun '19

18 Jun '19

Hi all, There will be a NumPy Community meeting on June 12 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

(Value Based Promotion) Current Behaviour
by Sebastian Berg 15 Jun '19

15 Jun '19

4 7

Extent to which to work around matrix and other duck/subclass limitations
by Marten van Kerkwijk 15 Jun '19

15 Jun '19

Hi All, In https://github.com/numpy/numpy/pull/12801, Tyler has been trying to use the new `where` argument for reductions to implement `nansum`, etc., using simplifications that boil down to `np.sum(..., where=~isnan(...))`. A problem that occurs is that `np.sum` will use a `.sum` method if that is present, and for matrix, the `.sum` method does not take a `where` argument. Since the `where` argument has been introduced only recently, similar problems may happen for array mimics that implement their own `.sum` method. The question now is what to do, with options being: 1. Let's stick with the existing implementation; the speed-up is not that great anyway. 2. Use try/except around the new implementation and use the old one if it fails. 3. As (2), but emit a deprecation warning. This will help array mimics, but not matrix (unless someone makes a PR; would we even accept it?); 4. Use the new implementation. `matrix` should be gone anyway and array mimics can either update their `.sum()` method or override `np.nansum` with `__array_function__`. Personally, I'd prefer (4), but realize that (3) is probably the more safer approach, even if it is really annoying to still be taking into account matrix deficiencies. All the best, Marten p.s. One could also avoid the `.sum()` override altogether by doing `np.add.reduce(..., where=...)`, but this would probably break just as much.

7 22

Planning for 1.18
by Charles R Harris 14 Jun '19

14 Jun '19

Hi All, With the 1.17 branch coming soon, this might be a good time to make plans about 1.18 development. A couple of possibilities are: - Expiring old deprecations, - Removing Python 2.7 compatibility code, - Design of a masked array replacement. Those proposals are not earth shaking, I see 1.18 as a chance to let the big changes coming in 1.17 settle down. If you have more ideas, please add them to the list. Chuck

2 1

New release note strategy after branching 1.17.
by Sebastian Berg 13 Jun '19

13 Jun '19

Hi all, we had discussed trying a new strategy to gather release notes on the last community call, but not followed up on it on the list yet. For the next release, we decided to try a strategy of using a wiki page to gather release notes. The main purpose for this is to avoid merge conflicts in the release notes file. It may also make things slightly easier for new contributors even without merge conflicts. Any comments/opinions about other alternatives are welcome. We probably still need to fix some details, but I this will probably mean: 1. We tag issues with "Needs Release Notes" 2. We ask contributors/maintainers to edit the initial PR post/comment with a release note snippet. (I expect maintainers may typically put in a placeholder as a start for the contributor.) 3. After merging, the release notes are copied into the wiki by the user or a contributor. After the copy happened, the label could/should be removed? SciPy uses a similar strategy, so they may already have some experience to do it slightly different that I am missing. Best Regards, Sebastian

8 11

branch 1.17
by Charles R Harris 12 Jun '19

12 Jun '19

Hi All, I'm thinking of branching 1.17 this coming (June 15) weekend. The major components are in: pocketfft, randomgen, and the new sorting routines, along with default `__array_function__`. They may need a bit of polish, but I think little remains to be done. If there are other pending PRs that you think need to be in the next release, please yell. This would also be a good time to review the release notes and go through the 1.17 issues to determine if any are release blockers. Cheers, Chuck

1 0

NumPy Community Meeting June 12
by Tyler Reddy 11 Jun '19

11 Jun '19

Hi, There will be a NumPy Community meeting on June 12 at 11 am Pacific Time. Anyone is free to join and edit the work-in-progress meeting notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?view Best wishes, Tyler

1 0