memory error caused by astype()
![](https://secure.gravatar.com/avatar/90db1b07cb3334aa7a5dc39ecbb30e1c.jpg?s=120&d=mm&r=g)
Hi list, When I do large array manipulations, I get out-of-memory errors. If the array size is 5000 by 6000, the following codes use nearly 1G. Then my PC displays a Python error box. The try/except won't catch it if the memory error happens in "astype" instead of "array1* array2" try: if ( array1.typecode() in cplx_types ): array1 = abs(array1.astype(Numeric.Complex32)) else: array1 = array1.astype(Numeric.Float32) if ( array2.typecode() in cplx_types ): array2 = abs(array2.astype(Numeric.Complex32)) else: array2 = array2.astype(Numeric.Float32) array1 = Numeric.sqrt(array1) * Numeric.sqrt(array2) return array1 except: gvutils.error("Memory error occurred\nPlease select a smaller array") return None My questions are: 1) Is there a more memory efficient way of doing this? 2) How do I deal with exception if astype is the only way to go 3) Is there a way in Python that detects the available RAM and limits the array size before he/she can go ahead with the array multiplications? i.e. detects the available RAM, say 800K Assume worst case - Complex32 Figure out how many temp_arrays used by numpy Calculate array size limit = ?? 4) If there is no 3) Is there something in Python that monitors memory and warns the user. I have these "astype" at a number functions. Do I have to put try/except at each location? Thanks, Shaw Gong
![](https://secure.gravatar.com/avatar/90db1b07cb3334aa7a5dc39ecbb30e1c.jpg?s=120&d=mm&r=g)
Hi list, When I do large array manipulations, I get out-of-memory errors. For instance if the array size is 5000 by 6000, the following codes use nearly 1G of RAM. Then my PC displays a Python error box. The try/except won't even catch it if the error happens in "astype" instead of "array1* array2" try: if ( array1.typecode() in cplx_types ): array1 = abs(array1.astype(Numeric.Complex32)) else: array1 = array1.astype(Numeric.Float32) if ( array2.typecode() in cplx_types ): array2 = abs(array2.astype(Numeric.Complex32)) else: array2 = array2.astype(Numeric.Float32) array1 = Numeric.sqrt(array1) * Numeric.sqrt(array2) return array1 except: gvutils.error("Memory error occurred\nPlease select a smaller array") return None My questions are: 1) Is there a more memory efficient way instead of using astype? 2) If not, then how do I catch error during astype? 3) Is there a way in Python that detects the available RAM and limits the array size before he/she can go ahead with the array multiplications? i.e. detects the available RAM, say 1G Assume the worst case - Complex32 Figure out the array size limit and warn user Thanks, Shaw Gong
participants (1)
-
Gong, Shawn (Contractor)