Debugging reason for python running unreasonably slow when adding numbers
Oscar Benjamin
oscar.j.benjamin at gmail.com
Tue Mar 14 18:55:33 EDT 2023
On Tue, 14 Mar 2023 at 16:27, Alexander Nestorov <alexandernst at gmail.com> wrote:
>
> I'm working on an NLP and I got bitten by an unreasonably slow behaviour in Python while operating with small amounts of numbers.
>
> I have the following code:
>
> ```python
> import random, time
> from functools import reduce
>
> def trainPerceptron(perceptron, data):
> learningRate = 0.002
> weights = perceptron['weights']
> error = 0
> for chunk in data:
> input = chunk['input']
> output = chunk['output']
>
> # 12x slower than equivalent JS
> sum_ = 0
> for key in input:
> v = weights[key]
> sum_ += v
In Python something like your task here would usually use something
along the lines of NumPy. Your two innermost loops involve adding up a
subset of numbers from a list chosen using a list of indices. This is
something that numpy can do much more efficiently with its fancy
indexing e.g.:
In [3]: a = np.array([1, 2, 3, 4, 5, 6, 7])
In [4]: b = np.array([0, 3, 5])
In [5]: a[b]
Out[5]: array([1, 4, 6])
In [6]: a[b].sum()
Out[6]: 11
This a[b].sum() operation in your code would be weights[input].sum()
and would be much faster than the loop shown (the speed difference
will be larger if you increase the size of the input array).
--
Oscar
More information about the Python-list
mailing list