Cluster imbalance caused due to #Num_Tokens

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cluster imbalance caused due to #Num_Tokens

Tiwari, Tarun

Hi,

 

While setting up a cluster for our POC, when we installed Cassandra on the 1st node we gave num_tokens: 256 , while on next 2 nodes which were added later we left it blank in Cassandra.yaml.

 

This made our cluster an unbalanced one with nodetool status showing 99% load on one server. Now even if I am setting up num tokens in the other 2 nodes as 256, its not seeming to effect. The wiki article  http://wiki.apache.org/cassandra/VirtualNodes/Balance doesn’t seem to provide steps to correct from this situation.

 

I read that there was nodetool balance kind of command in Cassandra 0.7 but not anymore.

 

UN  Node3  23.72 MB   1       0.4%   41a71df-7e6c-40ab-902f-237697eaaf3e  rack1

UN  Node2  79.35 MB   1       0.5%   98c493b-f661-491e-9d1f-1803f859528b  rack1

UN  Node1  86.93 MB   256     99.1%  a35ccca-556c-4f77-aa6d-7e3dad41ecf8  rack1

 

Is there something that we can do now balance the cluster?

 

Regards,

Tarun

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cluster imbalance caused due to #Num_Tokens

Robert Coli-3
On Tue, Apr 21, 2015 at 10:14 PM, Tiwari, Tarun <[hidden email]> wrote:

I read that there was nodetool balance kind of command in Cassandra 0.7 but not anymore.


It never worked, really.
 

UN  Node3  23.72 MB   1       0.4%   41a71df-7e6c-40ab-902f-237697eaaf3e  rack1

UN  Node2  79.35 MB   1       0.5%   98c493b-f661-491e-9d1f-1803f859528b  rack1

UN  Node1  86.93 MB   256     99.1%  a35ccca-556c-4f77-aa6d-7e3dad41ecf8  rack1

 

Is there something that we can do now balance the cluster?


Just run removenode on the two nodes with almost no data, one at a time.

Then add them back with vnodes enabled, one at a time.

As an aside, if the size of this cluster will never be greater than 9 or so nodes, you Mostly Lose by using vnodes.

=Rob
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cluster imbalance caused due to #Num_Tokens

Kiran mk

Bring down the second node using nodetool removenode or decommission

Add the node with num_tokens and run the nodetool repair. 

At last run the nodetool cleanup on both the nodes (one after the other)

Observe after some time using nodetool status.

On Apr 23, 2015 12:39 AM, "Robert Coli" <[hidden email]> wrote:
On Tue, Apr 21, 2015 at 10:14 PM, Tiwari, Tarun <[hidden email]> wrote:

I read that there was nodetool balance kind of command in Cassandra 0.7 but not anymore.


It never worked, really.
 

UN  Node3  23.72 MB   1       0.4%   41a71df-7e6c-40ab-902f-237697eaaf3e  rack1

UN  Node2  79.35 MB   1       0.5%   98c493b-f661-491e-9d1f-1803f859528b  rack1

UN  Node1  86.93 MB   256     99.1%  a35ccca-556c-4f77-aa6d-7e3dad41ecf8  rack1

 

Is there something that we can do now balance the cluster?


Just run removenode on the two nodes with almost no data, one at a time.

Then add them back with vnodes enabled, one at a time.

As an aside, if the size of this cluster will never be greater than 9 or so nodes, you Mostly Lose by using vnodes.

=Rob
 
Loading...