Hi everyone, I know had raised these questions in the PR, but wanted to post them in the mailing list as well. 1) Once NumPy adds the framework and initial set of Universal Intrinsic, if contributors want to leverage a new architecture specific SIMD instruction, will they be expected to add software implementation of this instruction for all other architectures too? 2) On whom does the burden lie to ensure that new implementations are benchmarked and shows benefits on every architecture? What happens if optimizing an Ufunc leads to improving performance on one architecture and worsens performance on another? Thanks, Raghuveer -----Original Message----- From: NumPy-Discussion On Behalf Of Daniele Nicolodi Sent: Tuesday, February 4, 2020 10:01 AM To: numpy-discussion@python.org Subject: Re: [Numpy-discussion] NEP 38 - Universal SIMD intrinsics On 04-02-2020 08:08, Matti Picus wrote:

Together with Sayed Adel (cc) and Ralf, I am pleased to put the draft version of NEP 38 [0] up for discussion. As per NEP 0, this is the next step in the community accepting the approach layed out in the NEP. The NEP PR [1] has already garnered a fair amount of discussion about the viability of Universal SIMD Intrinsics, so I will try to capture some of that here as well.

Hello, more interesting prior art may be found in VOLK https://www.libvolk.org. VOLK is developed mainly to be used in GNURadio, and this reflects in the available kernels and in the supported data types, I think the approach used there may be of interest. Cheers, Dan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Tue, Feb 4, 2020 at 2:00 PM Hameer Abbasi wrote:

—snip—

...

1) Once NumPy adds the framework and initial set of Universal Intrinsic, if contributors want to leverage a new architecture specific SIMD instruction, will they be expected to add software implementation of this instruction for all other architectures too?

In my opinion, if the instructions are lower, then yes. For example, one cannot add AVX-512 without adding, for example adding AVX-256 and AVX-128 and SSE*. However, I would not expect one person or team to be an expert in all assemblies, so intrinsics for one architecture can be developed independently of another.

I think this doesn't quite answer the question. If I understand correctly, it's about a single instruction (e.g. one needs "VEXP2PD" and it's missing from the supported AVX512 instructions in master). I think the answer is yes, it needs to be added for other architectures as well. Otherwise, if universal intrinsics are added ad-hoc and there's no guarantee that a universal instruction is available for all main supported platforms, then over time there won't be much that's "universal" about the framework. This is a different question though from adding a new ufunc implementation. I would expect accelerating ufuncs via intrinsics that are already supported to be much more common than having to add new intrinsics. Does that sound right?

...

2) On whom does the burden lie to ensure that new implementations are benchmarked and shows benefits on every architecture? What happens if optimizing an Ufunc leads to improving performance on one architecture and worsens performance on another?

This is slightly hard to provide a recipe for. I suspect it may take a while before this becomes an issue, since we don't have much SIMD code to begin with. So adding new code with benchmarks will likely show improvements on all architectures (we should ensure benchmarks can be run via CI, otherwise it's too onerous). And if not and it's not easily fixable, the problematic platform could be skipped so performance there is unchanged. Only once there's existing universal intrinsics and then they're tweaked will we have to be much more careful I'd think. Cheers, Ralf

I would look at this from a maintainability point of view. If we are increasing the code size by 20% for a certain ufunc, there must be a domonstrable 20% increase in performance on any CPU. That is to say, micro-optimisation will be unwelcome, and code readability will be preferable. Usually we ask the submitter of the PR to test the PR with a machine they have on hand, and I would be inclined to keep this trend of self-reporting. Of course, if someone else came along and reported a performance regression of, say, 10%, then we have increased code by 20%, with only a net 5% gain in performance, and the PR will have to be reverted.

—snip— _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

...

I think this doesn't quite answer the question. If I understand correctly, it's about a single instruction (e.g. one needs "VEXP2PD" and it's missing from the supported AVX512 instructions in master). I think the answer is yes, it needs to be added for other architectures as well.

That adds a lot of overhead to write SIMD based optimizations which can discourage contributors. It’s also an unreasonable expectation that a developer be familiar with SIMD of all the architectures. On top of that the performance implications aren’t clear. Software implementations of hardware instructions might perform worse and might not even produce the same result. From: NumPy-Discussion On Behalf Of Ralf Gommers Sent: Monday, February 10, 2020 9:17 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] NEP 38 - Universal SIMD intrinsics On Tue, Feb 4, 2020 at 2:00 PM Hameer Abbasi mailto:einstein.edison@gmail.com> wrote: —snip—

1) Once NumPy adds the framework and initial set of Universal Intrinsic, if contributors want to leverage a new architecture specific SIMD instruction, will they be expected to add software implementation of this instruction for all other architectures too?

In my opinion, if the instructions are lower, then yes. For example, one cannot add AVX-512 without adding, for example adding AVX-256 and AVX-128 and SSE*. However, I would not expect one person or team to be an expert in all assemblies, so intrinsics for one architecture can be developed independently of another. I think this doesn't quite answer the question. If I understand correctly, it's about a single instruction (e.g. one needs "VEXP2PD" and it's missing from the supported AVX512 instructions in master). I think the answer is yes, it needs to be added for other architectures as well. Otherwise, if universal intrinsics are added ad-hoc and there's no guarantee that a universal instruction is available for all main supported platforms, then over time there won't be much that's "universal" about the framework. This is a different question though from adding a new ufunc implementation. I would expect accelerating ufuncs via intrinsics that are already supported to be much more common than having to add new intrinsics. Does that sound right?

2) On whom does the burden lie to ensure that new implementations are benchmarked and shows benefits on every architecture? What happens if optimizing an Ufunc leads to improving performance on one architecture and worsens performance on another?

This is slightly hard to provide a recipe for. I suspect it may take a while before this becomes an issue, since we don't have much SIMD code to begin with. So adding new code with benchmarks will likely show improvements on all architectures (we should ensure benchmarks can be run via CI, otherwise it's too onerous). And if not and it's not easily fixable, the problematic platform could be skipped so performance there is unchanged. Only once there's existing universal intrinsics and then they're tweaked will we have to be much more careful I'd think. Cheers, Ralf I would look at this from a maintainability point of view. If we are increasing the code size by 20% for a certain ufunc, there must be a domonstrable 20% increase in performance on any CPU. That is to say, micro-optimisation will be unwelcome, and code readability will be preferable. Usually we ask the submitter of the PR to test the PR with a machine they have on hand, and I would be inclined to keep this trend of self-reporting. Of course, if someone else came along and reported a performance regression of, say, 10%, then we have increased code by 20%, with only a net 5% gain in performance, and the PR will have to be reverted. —snip— _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.orgmailto:NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On 11/2/20 8:02 pm, Devulapalli, Raghuveer wrote:

On top of that the performance implications aren’t clear. Software implementations of hardware instructions might perform worse and might not even produce the same result.

The proposal for universal intrinsics does not enable replacing an intrinsic on one platform with a software emulation on another: the intrinsics are meant to be compile-time defines that overlay the universal intrinsic with a platform specific one. In order to use a new intrinsic, it must have parallel intrinsics on the other platforms, or cannot be used there: "NPY_CPU_HAVE(FEATURE_NAME)" will always return false so the compiler will not even build a loop for that platform. I will try to clarify that intention in the NEP. I hope there will not be a demand to use many non-universal intrinsics in ufuncs, we will need to work this out on a case-by-case basis in each ufunc. Does that sound reasonable? Are there intrinsics you have already used that have no parallel on other platforms? Matti

...

I hope there will not be a demand to use many non-universal intrinsics in ufuncs, we will need to work this out on a case-by-case basis in each ufunc. Does that sound reasonable? Are there intrinsics you have already used that have no parallel on other platforms?

I think that is reasonable. It's hard to anticipate the future need and benefit of specialized intrinsics but I tried to make a list of some of the specialized intrinsics that are currently in use in NumPy that I don’t believe exist on other platforms (most of these actually don’t exist on AVX2 either). I am not an expert in ARM or VSX architecture, so please correct me if I am wrong. a. _mm512_mask_i32gather_ps b. _mm512_mask_i32scatter_ps/_mm512_mask_i32scatter_pd c. _mm512_maskz_loadu_pd/_mm512_maskz_loadu_ps d. _mm512_getexp_ps e. _mm512_getmant_ps f. _mm512_scalef_ps g. _mm512_permutex2var_ps, _mm512_permutex2var_pd h. _mm512_maskz_div_ps, _mm512_maskz_div_pd i. _mm512_permute_ps/_mm512_permute_pd j. _mm512_sqrt_ps/pd (I could be wrong on this one, but from the little google search I did, it seems like power ISA doesn’t have a vectorized sqrt instruction) Software implementations of these instructions is definitely possible. But some of them are not trivial to implement and are surely not going to be one line macro's either. I am also unsure of what implications this has on performance, but we will hopefully find out once we convert these to universal intrinsic and then benchmark. Raghuveer -----Original Message----- From: NumPy-Discussion On Behalf Of Matti Picus Sent: Tuesday, February 11, 2020 11:19 PM To: numpy-discussion@python.org Subject: Re: [Numpy-discussion] NEP 38 - Universal SIMD intrinsics On 11/2/20 8:02 pm, Devulapalli, Raghuveer wrote:

On top of that the performance implications aren’t clear. Software implementations of hardware instructions might perform worse and might not even produce the same result.

The proposal for universal intrinsics does not enable replacing an intrinsic on one platform with a software emulation on another: the intrinsics are meant to be compile-time defines that overlay the universal intrinsic with a platform specific one. In order to use a new intrinsic, it must have parallel intrinsics on the other platforms, or cannot be used there: "NPY_CPU_HAVE(FEATURE_NAME)" will always return false so the compiler will not even build a loop for that platform. I will try to clarify that intention in the NEP. I hope there will not be a demand to use many non-universal intrinsics in ufuncs, we will need to work this out on a case-by-case basis in each ufunc. Does that sound reasonable? Are there intrinsics you have already used that have no parallel on other platforms? Matti _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Wed, Feb 12, 2020 at 1:37 PM Devulapalli, Raghuveer < raghuveer.devulapalli@intel.com> wrote:

...

...

I hope there will not be a demand to use many non-universal intrinsics in ufuncs, we will need to work this out on a case-by-case basis in each ufunc. Does that sound reasonable? Are there intrinsics you have already used that have no parallel on other platforms?

I think that is reasonable. It's hard to anticipate the future need and benefit of specialized intrinsics but I tried to make a list of some of the specialized intrinsics that are currently in use in NumPy that I don’t believe exist on other platforms (most of these actually don’t exist on AVX2 either). I am not an expert in ARM or VSX architecture, so please correct me if I am wrong.

a. _mm512_mask_i32gather_ps b. _mm512_mask_i32scatter_ps/_mm512_mask_i32scatter_pd c. _mm512_maskz_loadu_pd/_mm512_maskz_loadu_ps d. _mm512_getexp_ps e. _mm512_getmant_ps f. _mm512_scalef_ps g. _mm512_permutex2var_ps, _mm512_permutex2var_pd h. _mm512_maskz_div_ps, _mm512_maskz_div_pd i. _mm512_permute_ps/_mm512_permute_pd j. _mm512_sqrt_ps/pd (I could be wrong on this one, but from the little google search I did, it seems like power ISA doesn’t have a vectorized sqrt instruction)

Software implementations of these instructions is definitely possible. But some of them are not trivial to implement and are surely not going to be one line macro's either. I am also unsure of what implications this has on performance, but we will hopefully find out once we convert these to universal intrinsic and then benchmark.

For these it seems like we don't want software implementations of the universal intrinsics - if there's no equivalent on PPC/ARM and there's enough value (performance gain given additional code complexity) in the additional AVX instructions, then we should still simply use AVX instructions directly. Ralf

Raghuveer

-----Original Message----- From: NumPy-Discussion On Behalf Of Matti Picus Sent: Tuesday, February 11, 2020 11:19 PM To: numpy-discussion@python.org Subject: Re: [Numpy-discussion] NEP 38 - Universal SIMD intrinsics

On 11/2/20 8:02 pm, Devulapalli, Raghuveer wrote:

...

On top of that the performance implications aren’t clear. Software implementations of hardware instructions might perform worse and might not even produce the same result.

The proposal for universal intrinsics does not enable replacing an intrinsic on one platform with a software emulation on another: the intrinsics are meant to be compile-time defines that overlay the universal intrinsic with a platform specific one. In order to use a new intrinsic, it must have parallel intrinsics on the other platforms, or cannot be used there: "NPY_CPU_HAVE(FEATURE_NAME)" will always return false so the compiler will not even build a loop for that platform. I will try to clarify that intention in the NEP.

I hope there will not be a demand to use many non-universal intrinsics in ufuncs, we will need to work this out on a case-by-case basis in each ufunc. Does that sound reasonable? Are there intrinsics you have already used that have no parallel on other platforms?

Matti

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion