From jlaura at asu.edu Wed Jun 5 16:30:51 2013 From: jlaura at asu.edu (Jay L.) Date: Wed, 5 Jun 2013 07:30:51 -0700 Subject: [concurrency] map error returning object Message-ID: List, I am working with the multiprocessing module attempting to get parallel reads working. Currently, if I open the file in each child process and read from an offset I achieve concurrency. If I attempt to open the file in the parent (via a class) and pass the class object to the children I get an overflow error. Here is a snippet: def get_object(row): return file.get(row) #Where get seeks to the offset and returns the data object. rows = range(len(file)) #This gets the row ids. pool = mp.Pool() results = pool.map(get_object, rows)print results #Should be a list of objects, so not human readable The error: OverflowError: Python int too large to convert to C long The error is being generated in line 528 of pool.py (Enthought Python 7.3, which is Python 2.7). Digging into that source this line is the final line in the get method of the ApplyResult class: def get(self, timeout=None) self.wait(timeout) if not self._ready: raise TimeoutError if self._success: return self._value else: raise self._value Am I correct in interpreting this to mean that the code is passing the first self._ready check, but the failing the self._success check (that appears to be just another self._ready check wrapped in the successful function) def successful(self): assert self._ready return self._success Thanks for any info or insight. The error is quite cryptic (to me) as I would expect to see if I was misusing range or working with gigantic numbers, not class objects. Jay -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Wed Jun 5 18:07:42 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 05 Jun 2013 17:07:42 +0100 Subject: [concurrency] map error returning object In-Reply-To: References: Message-ID: <51AF624E.5010606@gmail.com> On 05/06/2013 3:30pm, Jay L. wrote: > I am working with the multiprocessing module attempting to get parallel > reads working. Currently, if I open the file in each child process and > read from an offset I achieve concurrency. If I attempt to open the > file in the parent (via a class) and pass the class object to the > children I get an overflow error. If you inherit the parent process's fd, then (if I remember correctly) all the processes will share the same file offset: seeking in one process will affect the file offset in *all* processes. It's not surprising that this causes problems, although I can't give a precise explanation of what is happening. BTW, the exception object returned to the main process will not have a traceback attached to it. To see a full traceback you could catch, print and reraise the exception in the worker process: def get_object(row): try: return file.get(row) except: # print exception with traceback then reraise sys.excepthook(**sys.exc_info()) raise From jlaura at asu.edu Wed Jun 5 18:42:50 2013 From: jlaura at asu.edu (Jay L.) Date: Wed, 5 Jun 2013 09:42:50 -0700 Subject: [concurrency] map error returning object In-Reply-To: <51AF624E.5010606@gmail.com> References: <51AF624E.5010606@gmail.com> Message-ID: Richard, I had not realized that the file descriptor was inherited by the child. It looks like PEP443 seeks to overcome this with an optional flag in Python 3.3. I wonder if something similar does not exist for those of us 'stuck' in 2.x for a bit longer. If anyone else has any ideas on the specifics of what is occurring, I would love to understand the lower level details of what the concurrent seeks/tells are doing to cause an overflow. Thanks, Jay P.S. Many thanks for the traceback code. This will likely become standard in a good bit of my test code as debugging can sometimes be a pain. On Wed, Jun 5, 2013 at 9:07 AM, Richard Oudkerk wrote: > On 05/06/2013 3:30pm, Jay L. wrote: > >> I am working with the multiprocessing module attempting to get parallel >> reads working. Currently, if I open the file in each child process and >> read from an offset I achieve concurrency. If I attempt to open the >> file in the parent (via a class) and pass the class object to the >> children I get an overflow error. >> > > If you inherit the parent process's fd, then (if I remember correctly) all > the processes will share the same file offset: seeking in one process will > affect the file offset in *all* processes. It's not surprising that this > causes problems, although I can't give a precise explanation of what is > happening. > > BTW, the exception object returned to the main process will not have a > traceback attached to it. To see a full traceback you could catch, print > and reraise the exception in the worker process: > > def get_object(row): > try: > return file.get(row) > except: > # print exception with traceback then reraise > sys.excepthook(**sys.exc_info(**)) > raise > ______________________________**_________________ > concurrency-sig mailing list > concurrency-sig at python.org > http://mail.python.org/**mailman/listinfo/concurrency-**sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Wed Jun 5 19:34:08 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 05 Jun 2013 18:34:08 +0100 Subject: [concurrency] map error returning object In-Reply-To: References: <51AF624E.5010606@gmail.com> Message-ID: <51AF7690.30808@gmail.com> On 05/06/2013 5:42pm, Jay L. wrote: > I had not realized that the file descriptor was inherited by the child. > It looks like PEP443 seeks > to overcome this with an optional flag in Python 3.3. I wonder if > something similar does not exist for those of us 'stuck' in 2.x for a > bit longer. PEP 433 would only effect processes started using fork+exec. Currently multiprocessing on Unix uses fork but not exec. And anyway, closing the fd would just mean that the child process would inherit a non-functional file object. From davidanderson2374 at outlook.com Wed Jun 5 13:34:46 2013 From: davidanderson2374 at outlook.com (David Anderson) Date: Wed, 5 Jun 2013 16:04:46 +0430 Subject: Janaúba - Aprovados lista publicada Message-ID: <1370448221714b3c9b4437c206bad4caed82963857@outlook.com> Jana?ba ANA ANGELICA PEREIRA ALVES, LETICIA BAIRLE, FRANCISCO ANDERSON VALE DO NASCIMENTO, PAULO DE SIQUEIRA SILVA, JO?O CARLOS MOREIRA DE CARVALHO, DAMIANA PEREIRA DE OLIVERIA, MARIA DO SOCORRO DE ALBURQUERQUE ARRUDA BARBOSA, JAIME CUSTODIO DA SILVA FILHO. SEBASTIANA M?RCIA GOMES DE MELO, ?RICA FRANCISCA BATISTA DE MELO, MAYRES RAQUEL DA SILVA PINHEIRO, JUCIMARA VICENTE DOS SANTOS, WEULLER TEIXEIRA DE MAGALHAES. Pontalina. Quixel? ANNIBERG CORDEIRO DE SOUZA SILVA, LUCAS FONTENELE SILVA DE CARVALHO, GERLANDE MARIA FERREIRA, RAFAELLA SAMPAIO DE ALENCAR, JO?O CARLOS MOREIRA DE CARVALHO, DENISE ARA?JO JUSTINO, MARIA NIVANEIDE DE ABREU LIMA, JOELMA C?NDIDO DA SILVA. TERESA RAQUEL DE MORAES ANDRADE, CARLA NAYANNE MOREIRA DE SOUZA, MAIRA NOGUEIRA ALVES, IAN VIEIRA LIMA AMORA DE SOUZA, ROBERTO ALMEIDA PIRES. Dois Irm?os do Buriti. Ch?cara ANA MARA DE SOUSA PEREIRA, LUANA FERNANDA FERNANDES ANDRADE, FRANCISCO ROGER GARCIA DE ALMEIDA, POLLYANNA DE O, JO?O CARLOS MOREIRA DE CARVALHO, DANIELLA FERNANDES DA SILVA, MARIA JOS? DA SILVA FERREIRA, J?SSICA VENTURA FREIRE. SOLANGE PAULINO CORREIA, BRUNA KETHEY DA SILVA PEIXOTO, LUIS HENRIQUE FREITAS GOMES, HELIO SOARES DE ARAUJO, RENER QUEIROZ VIEIRA. Dois Irm?os do Buriti.