Questions about bootrapping and compactions during bootstrapping

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Questions about bootrapping and compactions during bootstrapping

Donald Smith

Looking at the output of "nodetool netstats" I see that the bootstrapping nodes pulling from only two of the nine nodes currently in the datacenter.   That surprises me: I'd think the vnodes it pulls from would be randomly spread across the existing nodes.  We’re using Cassandra 2.0.11 with 256 vnodes each.

 

I also notice that while bootstrapping, the node is quite busy doing compactions.   There are over 1000 pending compactions on the new node and it’s not finished bootstrapping. I’d think those would be unnecessary, since the other nodes in the data center have zero pending compactions.  Perhaps the compactions explains why running “du –hs /var/lib/cassandra/data” on the new node shows more disk space usage than on the old nodes.

 

Is it reasonable to do “nodetool disableautocompaction” on the bootstrapping node? Should that be the default???

 

If I start bootstrapping one node, it's not yet in the cluster but it decides which token ranges it owns and requests streams for that data. If  I then try to bootstrap a SECOND node concurrently, it will take over ownership of some token ranges from the first node. Will the first node then adjust what data it streams?

 

It seems to me the cassandra server needs to keep track of both the OLD token ranges and vnodes and the NEW ones.  I’m not convinced that running two bootstraps concurrently (starting the second one after several minutes of delay) is safe.

 

Thanks, Don

 

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
[hidden email]


AudienceScience

 

Reply | Threaded
Open this post in threaded view
|

Re: Questions about bootrapping and compactions during bootstrapping

DuyHai Doan
"Is it reasonable to do “nodetool disableautocompaction” on the bootstrapping node?" --> It's a tricky question

By default compaction is here to guarantee that you don't have too many small SSTables hurting the read path. Now in production I've seen some people disabling temporarily auto compaction during bootstrap because the new node cannot keep up with compaction.

 But before disabling auto compaction I would advise to play with the streaming_throughput first. Lowering this threshold will give room for the new node to compact but will make the joining process last longer. Trade-off, as always.

On Wed, Dec 17, 2014 at 1:32 AM, Donald Smith <[hidden email]> wrote:

Looking at the output of "nodetool netstats" I see that the bootstrapping nodes pulling from only two of the nine nodes currently in the datacenter.   That surprises me: I'd think the vnodes it pulls from would be randomly spread across the existing nodes.  We’re using Cassandra 2.0.11 with 256 vnodes each.

 

I also notice that while bootstrapping, the node is quite busy doing compactions.   There are over 1000 pending compactions on the new node and it’s not finished bootstrapping. I’d think those would be unnecessary, since the other nodes in the data center have zero pending compactions.  Perhaps the compactions explains why running “du –hs /var/lib/cassandra/data” on the new node shows more disk space usage than on the old nodes.

 

Is it reasonable to do “nodetool disableautocompaction” on the bootstrapping node? Should that be the default???

 

If I start bootstrapping one node, it's not yet in the cluster but it decides which token ranges it owns and requests streams for that data. If  I then try to bootstrap a SECOND node concurrently, it will take over ownership of some token ranges from the first node. Will the first node then adjust what data it streams?

 

It seems to me the cassandra server needs to keep track of both the OLD token ranges and vnodes and the NEW ones.  I’m not convinced that running two bootstraps concurrently (starting the second one after several minutes of delay) is safe.

 

Thanks, Don

 

Donald A. Smith | Senior Software Engineer
P: <a href="tel:425.201.3900%20x%203866" value="+14252013900" target="_blank">425.201.3900 x 3866
C: <a href="tel:%28206%29%20819-5965" value="+12068195965" target="_blank">(206) 819-5965
F: <a href="tel:%28646%29%20443-2333" value="+16464432333" target="_blank">(646) 443-2333
[hidden email]


AudienceScience

 


Reply | Threaded
Open this post in threaded view
|

Re: Questions about bootrapping and compactions during bootstrapping

Robert Coli-3
In reply to this post by Donald Smith
On Tue, Dec 16, 2014 at 4:32 PM, Donald Smith <[hidden email]> wrote:

Is it reasonable to do “nodetool disableautocompaction” on the bootstrapping node? Should that be the default???


There's various current/recent JIRA about compaction vs. bootstrapping, esp. wrt LCS compaction. Search JIRA.
 

If I start bootstrapping one node, it's not yet in the cluster but it decides which token ranges it owns and requests streams for that data. If  I then try to bootstrap a SECOND node concurrently, it will take over ownership of some token ranges from the first node. Will the first node then adjust what data it streams?


and

Are most directly on point.

tl;dr - It depends on whether your version has the fix for 2434 and what "simultaneously" means. I would generally avoid it, but people have simultaneously bootstrapped in a world with 2434 for 5 years now and in practice they seem to prove what I call the "Coli Conjecture" :

"If you're storing your data in Cassandra, you probably don't actually care about durability or consistency, even if you think you do."

=Rob