Multiprocessing performance question
ijbrewster at alaska.edu
Mon Feb 18 23:34:28 EST 2019
> On Feb 18, 2019, at 6:37 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> I don't have anything to add regarding your experiments with
> multiprocessing, but:
> Israel Brewster <ijbrewster at alaska.edu> writes:
>> Which creates and populates an 800x1000 “grid” (represented as a flat
>> list at this point) of “boxes”, where a box is a
>> shapely.geometry.box(). This takes about 10 seconds to run.
> This seems like the kind of task NumPy <URL:http://www.numpy.org/> is
> designed to address: Generating and manipulating large-to-huge arrays of
> numbers, especially numbers that are representable directly in the
> machine's basic number types (such as moderate-scale integers).
> Have you tried using that library and timing the result?
Sort of. I am using that library, and in fact once I get the result I am converting it to a NumPy array for further use/processing, however I am still a NumPy newbie and have not been able to find a function that generates a numpy array from a function. There is the numpy.fromfunction() command, of course, but “…the function is called with … each parameter representing the coordinates of the array varying along a specific axis…”, which basically means (if my understanding/inital testing is correct) that my function would need to work with *arrays* of x,y coordinates. But the geometry.box() function needs individual x,y coordinates, not arrays, so I’d have to loop through the arrays and append to a new one or something to produce the output that numpy needs, which puts me back pretty much to the same code I already have.
There may be a way to make it work, but so far I haven’t been able to figure it out any better than the code I’ve got followed by converting to a numpy array. You do bring up a good point though: there is quite possibly a better way to do this, and knowing that would be just as good as knowing why multiprocessing doesn’t improve performance. Thanks!
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
> \ “You don't need a book of any description to help you have some |
> `\ kind of moral awareness.” —Dr. Francesca Stavrakoloulou, bible |
> _o__) scholar, 2011-05-08 |
> Ben Finney
More information about the Python-list