On Dec 29, 2010, at 10:43 AM, Martin v. Löwis wrote:

...

Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part.

The question really is whether you would be willing to port Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that this would be practically new code, but it would be much appreciated.

One question is whether it's actually feasible to implement that API with SysV semaphores.

...

...

I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux.

Hmmm, "preferred" is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC.

If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?)

Actually, I'd probably recommend the opposite. IMO POSIX IPC is better designed and easier to work with so it would be my first choice. SysV would be a fallback for when POSIX IPC is unavailable, broken, incomplete, etc. bye Philip

On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. Löwis" wrote:

...

...

I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC.

Under the condition that it is within reason to fix it before the release.

What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped.

Regards, Martin

The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. jesse

On Wed, Dec 29, 2010 at 1:34 PM, "Martin v. Löwis" wrote:

Am 29.12.2010 18:54, schrieb Jesse Noller:

...

On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. Löwis" wrote:

...

...

...

I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC.

Under the condition that it is within reason to fix it before the release.

What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped.

Regards, Martin

The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library.

I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place.

Also, what specific test are you referring to?

Regards, Martin

If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported.

If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported.

Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin

Am 29.12.2010 22:34, schrieb Jesse Noller:

On Dec 29, 2010, at 3:49 PM, "Martin v. Löwis" wrote:

...

...

If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported.

Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls.

So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better).

Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes.

So what do you propose to do? Regards, Martin

I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me.

And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker.

I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD.

Just skipping the test case seems inappropriate, when it really tests the core functionality of the module in question. The failing test really means that the module does not work on the system.

FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though.

I don't claim that it's a bug. I claim that FreeBSD is just not supported (saying that FreeBSD doesn't support the multiprocessing module might be more appropriate). Somebody will have to correctly fix it - either the FreeBSD people, or the multiprocessing authors, or the concurrent.future authors. Meanwhile, we must admit that it just doesn't work (or else the test case would not fail). Regards, Martin

On Wed, 29 Dec 2010 23:24:32 +0100 "Martin v. Löwis" wrote:

...

I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me.

And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker.

I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. So, IMO, skipping tests is enough. At worse, multiprocessing can issue a warning when imported under these OSes. Regards Antoine.

On Dec 29, 2010, at 5:46 PM, Nick Coghlan wrote:

On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou wrote:

...

On Wed, 29 Dec 2010 23:24:32 +0100 "Martin v. Löwis" wrote:

...

...

I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me.

And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker.

I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff.

Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with the arbitrary limits. So +1 for skipping the affected tests on FreeBSD so the buildbot state can tell us something useful (conditionally wrapping them with "expected failure" may be even better, as then we'll find out when the FreeBSD out-of-the-box configuration actually supports running them).

A bit of info about FreeBSD 7.2 and 8.0 (the only versions I have installed). My installs are set up explicitly for testing and so they have had very little customization. - `sysctl -a|grep p1003` displays the config values specific to POSIX IPC. I think `sysctl -a|grep ipc` shows values for SysV IPC. It looks like the POSIX 1003_1b names in sysctl are similar to the names here (e.g. SEM_NSEMS_MAX, SEM_VALUE_MAX, etc.) -- http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html - Under my 7.2, sysctl.p1003_1b.sem_nsems_max == 0 and I can't create semaphores with my posix_ipc extension. After `kldload sem`, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works even without `kldload sem`. `kldstat` doesn't show sem.ko as loaded, so I assume that POSIX semaphores are now a default part of the kernel. - Under my 7.2 *and* 8.0, sysctl.p1003_1b.mq_open_max == 0 and I can't create a message queue with my posix_ipc extension. After `kldload mqueuefs`, sysctl.p1003_1b.mq_open_max remains 0 (huh?) but my posix_ipc message queue demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max is read only even if I change it in /etc/sysctl.conf. According to this post, sem_nsems_max doesn't become dynamic until 8.1: http://www.gossamer-threads.com/lists/python/dev/875319?do=post_view_threade... Hope this helps a bit Philip

On 12/29/2010 5:44 PM, David Bolen wrote:

Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing.

That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release.

Testing on updated buildbots and documenting older BSD limits and fixes seems like the best solution. Martin is right that we really do want the tests to run to catch any real bugs (on our side). -- Terry Jan Reedy

On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote:

Le mercredi 29 décembre 2010 à 21:49 +0100, "Martin v. Löwis" a écrit :

...

Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls.

More numbers (on Linux):

- Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition)

FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more.

Great detective work! This would suggest that ProcessPoolExecutors are useable on FreeBSD 7.2 so long as the user doesn't create more than two at once (which probably isn't a big deal for most apps). So skipping the test is probably the way to go. Cheers, Brian

Victor

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com

Wiadomość napisana przez Martin v. Löwis w dniu 2010-12-30, o godz. 10:16:

Am 30.12.2010 04:45, schrieb Brian Quinlan:

...

So skipping the test is probably the way to go.

I'm still -1 on that proposal.

I agree with Martin, explanation follows. In general, I'm trying to think as the user: someone wrote a program and said it requires Python 3.2. I expect it to work on Python 3.2 available on my platform. Doesn't matter if it's bundled with the OS or custom built. When a module is not available, this should fail loudly and as fast as possible. This is currently the case with conditionally built modules like curses, sqlite3, tkinter and others. From the user's perspective: the import fails. From the administrator's perspective: the build reports skipped modules. The admin can then alter the environment to make the build behave as expected, and then run the tests to confirm it works. From an OS maintainer perspective it's even better: a module can be made available optionally with explicitly set dependencies and required configuration. In our specific case a bunch of things should be determined first: 1. Does it still fail on FreeBSD 7.3+? 2. Why is the semaphore limit so low in the first place? 3. What is a reasonable number for that limit? That should include situations where people run multiple applications. 4. What other configuration should be done to ensure the module won't break on runtime. Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. One problem is that the administrator can make the OS compatible/incompatible on runtime (with sysctl). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. Fortunately, there is visible action from Brian, Philip and David to make the problems go away. I'm sure it will be fixed pretty soon. Ultimately I also agree with Martin that an implementation using SysV IPC would probably be better from the user's perspective. As he said, even if the underlying code is not as pretty as with the POSIX APIs, once it's written and tested nobody cares because it's not exposed. However, it's far too late to change that now for 3.2. [1] On RedHat/CentOS the test freezes badly (issue 10517). -- Best regards, Łukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/

Am 30.12.2010 16:40, schrieb Antoine Pitrou:

On Thu, 30 Dec 2010 16:04:16 +0100 Łukasz Langa wrote:

...

Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work.

I really don't think it is our job to maintain a list of OS/versions which work and don't work.

Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say).

...

ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1].

I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine.

Well, "the culprit" really is FreeBSD. However, concurrent.futures apparently makes freehanded use of semaphores, in a way that the FreeBSD authors didn't expect them to be used (if they expected them to be used at all, that is). Regards, Martin

And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API?

Yes. Importing concurrent.futures.process should fail. Unfortunately, it's imported from __init__.py, so either we change the API to move the executors to the submodules, or we let creation of process pools fail (rather than the import). Regards, Martin

On Thu, 30 Dec 2010 12:36:47 -0800 Ethan Furman wrote:

Martin v. Löwis wrote:

...

...

And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API?

Yes. Importing concurrent.futures.process should fail.

If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems.

Mind you, I have no idea how hard it would be to make this happen in the module in an intelligent way.

As I said, we can just emit a Warning instead of raising an exception. Regards Antoine.

Le jeudi 30 décembre 2010 à 17:05 +0100, "Martin v. Löwis" a écrit :

...

I really don't think it is our job to maintain a list of OS/versions which work and don't work.

Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say).

I added informations about FreeBSD, NetBSD, Darwin and OpenBSD to the issue #10348: http://bugs.python.org/issue10348#msg125042 The maximum number of POSIX semaphores can be read with sysctl: - FreeBSD: "p1003_1b.sem_nsems_max" - NetBSD: "kern.posix.semmax" - Darwin: "kern.posix.sem.max" - OpenBSD: (no support) Victor

1. Does it still fail on FreeBSD 7.3+?

Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable.

2. Why is the semaphore limit so low in the first place?

I don't know - (Free)BSD is in the tradition of disliking SysV inventions, and POSIX inventions unless they originate from FreeBSD. This is polemics, of course :-) In any case, SysV IPC doesn't seem much better. In SysV, you allocate semaphores in sets. In FreeBSD 8.1, in the standard setting, you can have up to 10 semaphore identifiers, with up to 60 semaphores each. That may sound reasonable, except that there is also a limit on the total number of semaphores of 60. So it seems that switching to SysV IPC wouldn't actually improve things (except that these are tunable parameters already, so changing them requires only a reboot, not a recompile of the kernel).

3. What is a reasonable number for that limit? That should include situations where people run multiple applications.

POSIX specifies that the minimum acceptable value for the SEM_NSEMS_MAX setting is _POSIX_SEM_NSEMS_MAX, which in turn must be at least 256. We could argue that we refuse to use POSIX semaphores if the system doesn't conform to POSIX, i.e. has semaphore limit of less than 256. Regards, Martin

BTW - can anyone contribute data points from other *BSDs?

I don't have an installation of OpenBSD, but... In FreeBSD, POSIX semaphores are implemented in sys/kern/uipc_sem.c. In http://www.openbsd.org/cgi-bin/cvsweb/src/sys/kern/ that file doesn't exist. Also, in FreeBSD's limits.h, _POSIX_SEM_NSEMS_MAX is defined (surprisingly to 256); in http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/include/limits.h?rev=1.... this constant doesn't appear. So ISTM that OpenBSD doesn't implement POSIX semaphores. IIUC, this means that the multiprocessing module won't be fully functional, and its tests (and the concurrent.futures tests) will be skipped. NetBSD apparently supports sem_open since 2.0: http://netbsd.gw.com/cgi-bin/man-cgi?sem_open++NetBSD-current According to http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/uipc_sem.c?rev=1.22&content-type=text/x-cvsweb-markup&only_with_tag=MAIN SEM_MAX is 128 since 2007, and dynamically adjustable (no reboot). Regards, Martin

On Dec 29, 2010, at 12:49 PM, Martin v. Löwis wrote:

...

If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported.

Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls.

I actually think that my tests may be overdone - in order to probe for specific race conditions they use a lot of locks to force calls to complete in a specific order. I'm thinking about pairing the tests down to only demonstrate basic correctness. This should fix the tests on FreeBSD and Windows. Then, when Python 3.2 is released, I can gradually introduce more comprehensive tests while ensuring that I keep the buildbots green on all supported platforms. Thoughts? Cheers, Brian

So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better).

Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com

On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote:

The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot.

If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. In other words, a user of concurrent.futures really needs to tweak their FreeBSD install to make in fully functional. There should be a way (through sysctl, presumably) to query the maximum number of semaphores and skip the relevant tests on that basis. -- R. David Murray www.bitdance.com

On Wed, 29 Dec 2010 14:14:03 -0500 "R. David Murray" wrote:

On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote:

...

The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot.

If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports.

Doesn't it suggest a possible resource leak somewhere? Or do the concurrent tests really rely on creating many semaphores? (as opposed to say 5 or 10 of them) Regards Antoine.