[Tutor] os.listdir blocks threads?

Hugo González Monteverde hugonz-lists at h-lab.net
Mon Oct 17 20:12:11 CEST 2005


Hi, I wrote a quick module for spawning a subprocess and run the 
function, just as an excercise, after I read your post...

It will spawn a subprocess, then pass the serialized return values to 
the parent. So, if you want to do:

lala = os.listdir(".")

you do

lala = fmg(os.listdir, ".")

This will not block, but it will run only on UNIX/Linux, unless there is 
a version of python that includes support for cygwin, which has fork() 
for Win32.

The module is below:

###################################################################

"""
fmg.py

As all threads will block on a call to some os functions, this implements a
function for doing thread-safe syscalls/C library functions using a child
process.

Availability: UNIX
"""


import os
import cPickle

def fork_me_gently(myfunct, *myargs):
     """fork_me_gently(callable, arguments) -> retval
     Make an arbitrary function pseudo thread-safe by running
     it in a child process. Return what the passed callable returns."""

     #prepare a pipe for child-parent communication
     r_end, w_end = os.pipe()

     pid = os.fork()

     if pid == 0: #child
         retval = myfunct(*myargs)

         #Serialize list for sending to pipe
         #dump() needs a file object, not a fd
         w_end_fo = os.fdopen(w_end, 'w')
         cPickle.dump(retval, w_end_fo)
         w_end_fo.close()
         # DIE, CHILD!
         raise SystemExit

     else: #parent
         # load() needs a file object
         r_end_fo = os.fdopen(r_end)
         retval = cPickle.load(r_end_fo)

         #clean up after yourself before continuing
         os.waitpid(pid, 0)

         return(retval)

fmg = fork_me_gently

if __name__ == "__main__":
     #temp = ts_listdir(".")
     temp = fmg(os.listdir, ".")
     for i in temp:
         print i





Wolfgang Braun wrote:
> Hello List,
> 
> 
> I try to read large directories off network shares (samba3,NT) with
> os.listdir(). The listdir() takes up to 5 minutes and blocks the rest of
> the program (gui refreshes, ...)
> 
> The idea now is to put the listdir call into a separate thread so the
> program can go on doing stuff (see below).
> 
> Unfortunately, as soon as I start a rdir instance the whole python
> process locks up and waits for the os.listdir to return.
> 
> Plan b was to popen('/bin/ls -1 '%dir) in the rdir thread which works
> better but is a bit kludgy (needs to work on NT, too) and the program
> cannot be shudown cleanly unless the rdir thread has finished.
> 
> 
> Obviously I'm doing something wrong? What would be the Right Way to
> handle this situation?
> 
> 
> Thanks,
> Wolfgang
> 
> 
> # --- skeleton listdir in a thread
> 
> class rdir(threading.Thread):
>     def __init__(self,dir,glob=None):
>         super(rdir,self).__init__()
>         self.dir=dir
>         self.content=()
> 
>     def start(self):
>         self.setDaemon(True)
>         super(dircache,self).start()
> 
>     def run(self):
>             self.content=os.listdir(self.dir)
> 	    # alternatively os.popen(' /bin/ls -1U '%dir)
>             # stuff to keep run() from returning
> 
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor


More information about the Tutor mailing list