Chaz. wrote:
I started out using Spread some time ago (more than 2 years ago). The implementation was limited to a hundred or so nodes (that is in the notes on the spread implementation). Secondly it isn't quite so lightweight as you think (I've measured the performance).
It is a very nice system but when it gets to 1000s of machines very little work has been done on solving many of the problems. My research on it goes back almost a decade starting out with Horus.
I must admit to not having attempted to scale it that far, but I was under the impression that only the more expensive delivery modes were that costly. But by the sounds of it, you don't need me to tell you that.
Actually I am part of the IRTF group on P2P, E2E and SAM. I know the approaches they are being tossed about. I have tried to implement some of them. I just am not of the opinion that smart people can't find solutions to tough problems.
Ok, in which case my apologies. My reading of your posts had lead me to believe, incorrectly, you may not be familiar with the various issues. In that case, you can (should) disregard most of it.
Is multicast or broadcast the right way? I don't know, but I do know that without trying we will never know. Having been part of the IETF
It's clearly right for some things - I'm just not sure how much bi-directional distribution would be helped by it, since you've got at some point to get the replies back.
community for a lot of years (I was part of the group that worked on SNMP v1 and the WinSock standard), I know that when the "pedal meets the metal" sometimes you discover interesting things.
I didn't realise winsock went near the IETF. You learn something new every day.
Knuth and his comments on early optimisation apply here. Have you tried it? You might be surprised.
I am sorry to say I don't know the paper or research you are referring to. Can you point me to some references?
Sorry, it's a phrase from Donald Knuth's (excellent) three-volume programming book, "The Art of Computer Programming". Highly recommended.
Thanks for the information. This is what makes me think that I want something based on UDP and not TCP! And if I can do RMT (or some variant of it) I might be able to get better performance. But, as I said it is the nice thing about not having someone telling me I need to get a product out the door tomorrow! I have time to experiment and learn.
When I wrote my reply I hadn't seen your comment on the app being distributed storage.
How many calls per second are you doing, and approximately what volume of data will each call exchange?
This is information I can't provide since the system I have designing has no equivalent in the marketplace today (either commercial or open source). All I know is that the first version of the system I built - using C/C++ and a traditional architecture (a few dozens of machines) was able to handle 200 transactions/minute (using SOAP). While there were some "short messages" (less than an normal MTU), I had quite a few that topped out 50K bytes and some up to 100Mbytes.
Oh. In which case more or less everything I wrote is useless!
Successful transmission is really the easy bit for multicast. There is IGMP snooping, IGMP querier misbehaviour, loss of forwarding on an upstream IGP flap, flooding issues due to global MSDP issues, and so forth.
I agree about the successful transmission. You've lost me on the IGMP part. Can you elaborate as to your thoughts?
Well, my experience of large multicast IPv4 networks is that short interruptions in multicast connectivity are not uncommon. There are a number of reasons for this, which can be broadly broken down into 1st hop and subsequent hop issues. Basically, in a routed-multicast environment, I've seen the subnet IGMP querier (normally the gateway) get pre-empted by badly configured or plain broken OS stacks (e.g. someone running Linux with the IGMPv3 early patches). I've also seen confusion for highly-available subnets (e.g. VRRPed networks) where the IGMP querier and the multicast designated forwarder are different. This can cause issues with the IGMP snooping on the downstream layer2 switches when the DF is no longer on the path which the layer2 snooping builds. You also get issues with upstream changes in the unicast routing topology affecting PIM. Most of these are only issues with routed multicast. Subnet-local is a lot simpler, though you do still need an IGMP querier and switches with IGMP snooping.
I do know I need EXACTLY-ONCE semantics but how and where I implement them is the unknown. When you use TCP you assume the network provides the bulk of the solution. I have been thinking that if I use a less reliable network - one with low overhead - that I can provide the server part to do the EXACTLY-ONCE piece.
As to why I need EXACTLY-ONCE, well if I have to store something I know I absolutely need to store it. I can't be in the position that I don't know it has been stored - it must be there.
Thanks for the great remarks....I look forward to reading more.
This makes a lot more sense now I know it's storage related. You're right, this is a tricky and uncommon problem. Let me see if I've got this right: You're building some kind of distributed storage service. Clients will access the storage by a "normal" protocol to one of the nodes. Reads from the store are relatively easy, but writes to the store will need to be distributed to all or a subset of the nodes. Obviously you'll have a mix of lots of small writes and some very large writes. Hmm. Are you envisioning that you might have >1 storage set on the nodes, and using a different multicast group per storage set to build optimal distribution? You might be able to perform some tricks depending on whether this service provides block- or filesystem-level semantics. If it's the latter, you could import some techniques from the distributed version control arena - broadly speaking, node+version number each file and "broadcast" (in the application sense) just the file + newnode+newversion to the other store nodes, and have them lock the local copy and initiate a pull from the updated node. For block-level storage, that's going to be a lot harder. For the multicast, something like NORM, which as you probably know is basically forward-error-corrected transmit channel with receiver-triggered re-transmits, would probably work. An implementation would likely be non-trivial, but a fascinating project.