freebsd and multiprocessing

[Philip Semanchuk]

On Mar 2, 2010, at 1:31 PM, Tim Arnold wrote:

> On Mar 2, 12:59 pm, Tim Arnold <a_j... at bellsouth.net> wrote:
>> On Mar 2, 11:52 am, Philip Semanchuk <phi... at semanchuk.com> wrote:
>>> On Mar 2, 2010, at 11:31 AM, Tim Arnold wrote:
>>
>>>> Hi,
>>>> I'm intending to use multiprocessing on a freebsd machine (6.3
>>>> release, quad core, 8cpus, amd64). I see in the doc that on this
>>>> platform I can't use synchronize:
>>
>>>> ImportError: This platform lacks a functioning sem_open
>>>> implementation, therefore, the required synchronization primitives
>>>> needed will not function, see issue 3770.
>>
>>>> As far as I can tell, I have no need to synchronize the  
>>>> processes--I
>>>> have several processes run separately and I need to know when  
>>>> they're
>>>> all finished; there's no communication between them and each owns  
>>>> its
>>>> own log file for output.
>>
>>>> Is anyone using multiprocessing on FreeBSD and run into any other
>>>> gotchas?
>>
>>> Hi Tim,
>>> I don't use multiprocessing but I've written two low-level IPC
>>> packages, one for SysV IPC and the other for POSIX IPC.
>>
>>> I think that multiprocessing prefers POSIX IPC (which is where
>>> sem_open() comes from). I don't know what it uses if that's not
>>> available, but SysV IPC seems a likely alternative. I must  
>>> emphasize,
>>> however, that that's a guess on my part.
>>
>>> FreeBSD didn't have POSIX IPC support until 7.0, and that was sort  
>>> of
>>> broken until 7.2. As it happens, I was testing my POSIX IPC code
>>> against 7.2 last night and it works just fine.
>>
>>> SysV IPC works under FreeBSD 6 (and perhaps earlier versions; 6 is  
>>> the
>>> oldest I've tested). ISTR that by default each message queue is
>>> limited to 2048 bytes in total size. 'sysctl kern.ipc' can probably
>>> tell you that and may even let you change it. Other than that I  
>>> can't
>>> think of any SysV limitations that might bite you.
>>
>>> HTH
>>> Philip
>>
>> Hi Philip,
>> Thanks for that information. I wish I could upgrade the machine to
>> 7.2! alas, out of my power.  I get the following results from sysctl:
>> % sysctl kern.ipc | grep msg
>> kern.ipc.msgseg: 2048
>> kern.ipc.msgssz: 8
>> kern.ipc.msgtql: 40
>> kern.ipc.msgmnb: 2048
>> kern.ipc.msgmni: 40
>> kern.ipc.msgmax: 16384
>>
>> I'll write some test programs using multiprocessing and see how they
>> go before committing to rewrite my current code. I've also been
>> looking at 'parallel python' although it may have the same issues.http://www.parallelpython.com/
>>
>> thanks again,
>> --Tim
>
> Well that didn't work out well. I can't import either Queue or Pool
> from multiprocessing, so I'm back to the drawing board. I'll see now
> how parallel python does on freebsd.


Sorry to hear that didn't work for you. Should you need to get down to  
the nuts & bolts level, my module for SysV IPC is here:
http://semanchuk.com/philip/sysv_ipc/


Good luck with Parallel Python,
Philip