Experience with multihoming cassandra?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Experience with multihoming cassandra?

Donald Smith

We have large boxes with 256G of RAM and SSDs.      From iostat, top, and sar we think the system has excess capacity.  Anyone have recommendations about multihoming cassandra on such a node (connecting it to multiple IPs and running multiple cassandras simultaneously)?      I’m skeptical, since Cassandra already has built-in multi-threading and since if the node went down multiple nodes would disappear.  We’re using C* version 2.0.9.

 

A google/bing search for  multihoming cassandra doesn’t turn much up.

 

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
[hidden email]


AudienceScience

 

Reply | Threaded
Open this post in threaded view
|

Re: Experience with multihoming cassandra?

Jared Biel
Doing this seems counter-productive to Cassandra's design/use-cases. It's best at home running on a large number of smaller servers rather than a small number of large servers. Also, as you said, you won't get any of the high availability benefits that it offers if you run multiple copies of Cassandra on the same box.


On 25 September 2014 16:58, Donald Smith <[hidden email]> wrote:

We have large boxes with 256G of RAM and SSDs.      From iostat, top, and sar we think the system has excess capacity.  Anyone have recommendations about multihoming cassandra on such a node (connecting it to multiple IPs and running multiple cassandras simultaneously)?      I’m skeptical, since Cassandra already has built-in multi-threading and since if the node went down multiple nodes would disappear.  We’re using C* version 2.0.9.

 

A google/bing search for  multihoming cassandra doesn’t turn much up.

 

Donald A. Smith | Senior Software Engineer
P: <a href="tel:425.201.3900%20x%203866" value="+14252013900" target="_blank">425.201.3900 x 3866
C: <a href="tel:%28206%29%20819-5965" value="+12068195965" target="_blank">(206) 819-5965
F: <a href="tel:%28646%29%20443-2333" value="+16464432333" target="_blank">(646) 443-2333
[hidden email]


AudienceScience

 


Reply | Threaded
Open this post in threaded view
|

Re: Experience with multihoming cassandra?

Ben Bromhead
I'm guessing your talking about multi-homing because you want to have multiple tenants (different apps/ teams etc) to make better use of resources ?

As Jared mentioned running multiple Cassandra processes on the same hardware that participate in the same cluster doesn't make much sense from a failure domain point of view (it could mess up how C* replicates with replicas for a key being potentially on the same physical server). 

As for splitting up a server for multi-tenancy purposes this then becomes a question of virtualisation as while there is some multi-tenant support in C* (auth, throttling per keyspace), it is fairly limited at best. 

There a whole range of options out there ranging from xen, vmware etc through to lightweight virtualisation like linux namespaces with cgroups, etc. 

I think Spotify run C* in production using namespaces with cgroup iirc and you could using something like docker to help manage this for you. 

Docker will also help with managing network addressing etc (the multi homed aspect). We've also had a lot of success running C* with docker (and previously SmartOS and solaris zones). Though you will be treading new / undocumented ground and thus expect to have to solve a few issues along the way.



On 26 September 2014 04:32, Jared Biel <[hidden email]> wrote:
Doing this seems counter-productive to Cassandra's design/use-cases. It's best at home running on a large number of smaller servers rather than a small number of large servers. Also, as you said, you won't get any of the high availability benefits that it offers if you run multiple copies of Cassandra on the same box.


On 25 September 2014 16:58, Donald Smith <[hidden email]> wrote:

We have large boxes with 256G of RAM and SSDs.      From iostat, top, and sar we think the system has excess capacity.  Anyone have recommendations about multihoming cassandra on such a node (connecting it to multiple IPs and running multiple cassandras simultaneously)?      I’m skeptical, since Cassandra already has built-in multi-threading and since if the node went down multiple nodes would disappear.  We’re using C* version 2.0.9.

 

A google/bing search for  multihoming cassandra doesn’t turn much up.

 

Donald A. Smith | Senior Software Engineer
P: <a href="tel:425.201.3900%20x%203866" value="+14252013900" target="_blank">425.201.3900 x 3866
C: <a href="tel:%28206%29%20819-5965" value="+12068195965" target="_blank">(206) 819-5965
F: <a href="tel:%28646%29%20443-2333" value="+16464432333" target="_blank">(646) 443-2333
[hidden email]


AudienceScience

 





--

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359