<div dir="ltr">> <span style="font-family:arial,sans-serif;font-size:13px">is that client is limited to send operations only and requests to HTTP(S) protocol only. Is that true? All other parts of the system can communicate with whatever protocols they like.</span><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Yes! </font><span style="font-family:arial,sans-serif">Every incoming requests are HTTP. </span></div><div><span style="font-family:arial,sans-serif"><br>
</span></div><div><span style="font-family:arial,sans-serif">There also need to be a long running process to hold all the fd pools for DB connection, server side events (</span><font face="arial, sans-serif">text/event-stream</font><span style="font-family:arial,sans-serif">) connections, iOS APNS push conn pools, XMPP connection, etc. And there may be multiple nodes holding these fd for long opening</span></div>
<div><span style="font-family:arial,sans-serif"><br></span></div><div><span style="font-family:arial,sans-serif">Example: </span></div><div><span style="font-family:arial,sans-serif"><br></span></div><div><span style="font-family:arial,sans-serif"> * client HTTP requests are handled and finished on NodeA</span></div>
<div><span style="font-family:arial,sans-serif"> * NodeA notifies NodeB for events, NodeB gather the event for statistical analystis by writing some data to DB</span></div><div><span style="font-family:arial,sans-serif"> * also NodeA notifies NodeC to Server-Side-Events to some other connection</span></div>
<div><span style="font-family:arial,sans-serif"> * next NodeA notifies NodeC to send out an iOS push message</span></div><div><span style="font-family:arial,sans-serif"> * last NodeA notify NodeD on which sends out an XMPP message.</span></div>
<div><span style="font-family:arial,sans-serif"><br></span></div><div><span style="font-family:arial,sans-serif">Stuff goes like that.</span><br></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">> </font><span style="font-family:arial,sans-serif;font-size:13px">XML-RPC is a bad way of operation and I am going to promote that belief.</span></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Completely agree. But currently Telegraphy is one and only one-of-a-kind. </font></div><div><font face="arial, sans-serif"><br></font></div>
<div><span style="font-family:arial,sans-serif">Other options like <a href="http://brubeck.io">brubeck.io</a> with mongrel2, </span><font face="arial, sans-serif">webalchemy, etc.</font><br></div><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif">> </font><span style="font-family:arial,sans-serif;font-size:13px">the generic clustering problem will include management DNS infrastructure</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><font face="arial, sans-serif">For a running cluster, yes. But Nodejs has its cluster batteries included so it's easier to bootstrap a cluster in that way than say, build with multiprocessing/</font><span style="font-family:arial,sans-serif;font-size:13px">billiard from scratch</span></div>
<div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Mar 18, 2014 at 3:12 PM, anatoly techtonik <span dir="ltr"><<a href="mailto:techtonik@gmail.com" target="_blank">techtonik@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div class="">On Tue, Mar 18, 2014 at 5:16 AM, est <span dir="ltr"><<a href="mailto:electronixtar@gmail.com" target="_blank">electronixtar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div>IPython.parallel <div><br></div>
<div><a href="http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing" target="_blank">http://ipython.org/ipython-doc/stable/install/install.html#dependencies-for-ipython-parallel-parallel-computing</a><br>
</div><div><br></div><div>It's based on ZeroMQ(PyZMQ), and the `ssh` command. I don't think that's lightweigh enough for busy web clusters.</div></div></blockquote><div><br></div></div><div>You will need to secure you web cluster computations anyway. SSH may be slower that HTTPS, I agree, but I'd still see the benchmarks. IPython is good for handling long processing tasks. For myriad of tiny code+data workers I'd choose Stackless. Not sure about the web server part.</div>
<div class="">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>By QMachine I assume that's </div>
<div><br></div><div><a href="https://github.com/wilkinson/qmachine" target="_blank">https://github.com/wilkinson/qmachine</a><br></div><div><br></div><div>For web server cluster it's really not a good idea to amplify HTTP requests. One client request amplifies several other HTTP requests on server clusters.</div>
</div></blockquote><div><br></div></div><div>Right. Because your workers are not trusted you need to distribute the load and validate results with multiple passes.</div><div class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div>What I propose is something like Zed Shawn's Mongrel2 project (<a href="http://mongrel2.org/" target="_blank">http://mongrel2.org/</a>), use a very lightweight server-side serialization protocol as cluster IPC, you can pass states/data between nodes (workers) easily. It should be agnostic to framework or libraries, the objective is to unite python modules in the realtime web world. Because for request-response web world, a synchronized gateway like WSGI is good enough, between each requests, <a href="https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale" target="_blank">share nothing</a>.</div>
<div><br></div><div>But for realtime web, server side state is very much required. There need to be a fd pool for DBs, external services, and stuff like Server-Side-Push technologies. </div></div></blockquote><div><br></div>
</div><div>"realtime web" is a very broad term. Need a more concise definition. I see only one difference in "web" over standard protocol - is that client is limited to send operations only and requests to HTTP(S) protocol only. Is that true? All other parts of the system can communicate with whatever protocols they like.</div>
<div> </div><div>So, to unify the network under some standard, we need common base. Stick to limitations of client to make all nodes work the same. Limit choice to bare minimum and extend where it is needed.</div><div class="">
<div><br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Let's assume the following scenario:</div>
<div><br></div><div>One user submits a blog, his follower gets browser/iOS/Android push notification. Because users are connected different nodes in one big cluster, we need some kind of mechanism to broadcast this message. </div>
<div><br></div><div>In such an architecture we can write simpler code like this:</div><div><br></div><div>from django.db.models.signals import post_save<br></div><div><br></div><div>@receiver(post_save, sender=BlogPostModel)<br>
</div><div>def my_handler(sender, **kwargs):<br></div><div> msg = "User X just posted a new blog, check it out at http://..."</div><div><div> browser_followers.send(msg)</div></div> ios_followers.send(msg)<div>
android_followers.send(msg)<br></div><div><br></div><div>Currently this library reall shines.</div><div><br></div><div><a href="https://pypi.python.org/pypi/telegraphy/" target="_blank">https://pypi.python.org/pypi/telegraphy/</a><br>
</div><div><br></div><div>Telegraphy architecture is like this:</div><div><br></div><div><img alt="Inline image 1"><br></div><div>
<br>
</div><div>What I propose is to merge Web-app part and the AutobahnPython Gateway part into <i>one</i> based on a community honored standard. </div></div></blockquote><div><br></div></div><div>Just a side note - XML-RPC is a bad way of operation and I am going to promote that belief.</div>
<div><br></div><div>The key component here that is not depicted is client limitations (able to only request events, and accept events after websocket connection is established with a single server). Channel description (WS, HTTP) are not informative in this regard to capture that limitation that this architecture should deal with.</div>
<div><br></div><div>When client (browser) establishes connection to HTTP site, can it open a websocket to the site in other domain? If no - then cross-domain interaction should also be included into problem description before unifying Django and Autobahn. If this limitation exists - the generic clustering problem will include management DNS infrastructure (to make sure client can send requests to any node in the cluster) or clustering will require frontends on servers to reroute requests on established websocket connections to appropriate cluster nodes.</div>
<div><br></div><div>Not sure I got the positioning of NodeJS cluster right, so feel free to fix that.</div></div></div></div>
</blockquote></div><br></div></div></div>