On 25.06.2015 11:35, Sturla Molden wrote:
Nathaniel Smith <njs@pobox.com> wrote:
Continuing my vague and uninformed impressions, I suspect that this would actually be relatively easy to fix by hooking the import system to do something more intelligent, like nominate one node as the leader and have it do the file lookups and then tell everyone else what it found (via the existing message-passing systems).
There are two known solutions. One is basically what you describe. The other, which at least works on IBM blue brain, is to import modules from a ramdisk. It seems to be sufficient to make sure whatever is serving the shared disk can deal with the 100k client DDoS.
Another way to solve this problem may be to use our eGenix PyRun which embeds modules right in the binary. As a result, all reading is done from the mmap'ed binary and automatically shared between processes by the OS: http://www.egenix.com/products/python/PyRun/ I don't know whether this actually works on an IBM Blue Brain with 100k clients - we are not fortunate enough to have access to one of those machines :-) Note: Even though the data reading is shared, the resulting code and modules objects are, of course, not shared, so you still have the overhead of using up memory for this, unless you init your process cluster using fork() after you've imported all necessary modules (then you benefit from the copy-on-write provided by the OS - code objects usually don't change after they have been created). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 25 2015)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2015-06-25: Released mxODBC 3.3.3 ... http://egenix.com/go79 2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015-07-20: EuroPython 2015, Bilbao, Spain ... 25 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/