Quantcast

Cassandra hanging in IntervalTree.comparePoints() and in CompactionController.maxPurgeableTimestamp()

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cassandra hanging in IntervalTree.comparePoints() and in CompactionController.maxPurgeableTimestamp()

Donald Smith

We deployed a brand new 13 node 2.1.4 C* cluster and used sstabloader to stream about 500GB into cassandra.   The streaming took less than a day but afterwards  pending compactions do not decrease.  The Cassandra nodes (which have about 500 pending compactions each) seem to spend most of their time in

 

    IntervalTree.comparePoints() and in

    CompactionController.maxPurgegableTimestamp()

 

(sometimes, too, in com.google.common.util.concurrent.Uninterruptibles.sleepUninteruptedly()).  That’s what Java VisualVM shows that they’re doing in the sampler.

 

Many of the nodes show 100% cpu usage per core.

 

Any idea what’s causing it to hang?

 

Might http://qnalist.com/questions/5818079/reasons-for-nodes-not-compacting  or https://issues.apache.org/jira/browse/CASSANDRA-8914  explain it?  Altering the table to use ,  'cold_reads_to_omit': 0.0  didn’t help.

 

  Thanks, Don

 

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
[hidden email]


AudienceScience

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cassandra hanging in IntervalTree.comparePoints() and in CompactionController.maxPurgeableTimestamp()

Robert Coli-3
On Wed, Apr 29, 2015 at 8:40 AM, Donald Smith <[hidden email]> wrote:

We deployed a brand new 13 node 2.1.4 C* cluster and used sstabloader to stream about 500GB into cassandra.   The streaming took less than a day but afterwards  pending compactions do not decrease.  The Cassandra nodes (which have about 500 pending compactions each) seem to spend most of their time in


Try with the just-released 2.1.5?

=Rob
 
Loading...