Quantcast

Huge number of sstables after adding server to existing cluster

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Huge number of sstables after adding server to existing cluster

Mantas Klasavičius

Hi, 

we are running test cassandra cluster of 8 nodes running in single DC using Simple snitch and DateTieredCompactionStrategy after adding new node(9th) to the cluster we see that number of sstables on newly joined server roughly equals to sum of all sstables on all servers in the cluster. and that number is huge as tens of thousands of sstables on newly added server.

Q1:is that what we should expect to happen?

Furthermore newly added server seems isn't overloaded, basically there are no pending/scheduled compactions but the number of sstables isn't decreasing.

Q2:what could be the reason of not reducing number of sstables?

Q3:what we need to do to reduce number of sstables per server?

Thanks for your help

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Robert Coli-3
On Fri, Apr 3, 2015 at 4:57 AM, Mantas Klasavičius <[hidden email]> wrote:

Q1:is that what we should expect to happen?

A known problem with the current streaming paradigm when combined with vnodes is that newly bootstrapped nodes do a bunch of compaction.
 

Q2:what could be the reason of not reducing number of sstables?

nodetool setcompactionthroughput 0 # note, if you don't have spare i/o, this could negatively affect service time
 

Q3:what we need to do to reduce number of sstables per server?

Make sure you're compacting faster than you're writing and wait.

=Rob
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Pranay Agarwal
I remember once that happening to me. The SStables were way beyond the limit (32 default) but the compaction were still not starting. All I did was "nodetool enableautocompaction keyspace table" and the compaction immediately started and count of SSTables were down to normal level. It was little surprising to me as well because I had never disabled compaction in the first place.

-Pranay 

On Fri, Apr 3, 2015 at 10:18 AM, Robert Coli <[hidden email]> wrote:
On Fri, Apr 3, 2015 at 4:57 AM, Mantas Klasavičius <[hidden email]> wrote:

Q1:is that what we should expect to happen?

A known problem with the current streaming paradigm when combined with vnodes is that newly bootstrapped nodes do a bunch of compaction.
 

Q2:what could be the reason of not reducing number of sstables?

nodetool setcompactionthroughput 0 # note, if you don't have spare i/o, this could negatively affect service time
 

Q3:what we need to do to reduce number of sstables per server?

Make sure you're compacting faster than you're writing and wait.

=Rob

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Thomas Borg Salling
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.
/Thomas.

2015-04-03 19:33 GMT+02:00 Pranay Agarwal <[hidden email]>:
I remember once that happening to me. The SStables were way beyond the limit (32 default) but the compaction were still not starting. All I did was "nodetool enableautocompaction keyspace table" and the compaction immediately started and count of SSTables were down to normal level. It was little surprising to me as well because I had never disabled compaction in the first place.

-Pranay 

On Fri, Apr 3, 2015 at 10:18 AM, Robert Coli <[hidden email]> wrote:
On Fri, Apr 3, 2015 at 4:57 AM, Mantas Klasavičius <[hidden email]> wrote:

Q1:is that what we should expect to happen?

A known problem with the current streaming paradigm when combined with vnodes is that newly bootstrapped nodes do a bunch of compaction.
 

Q2:what could be the reason of not reducing number of sstables?

nodetool setcompactionthroughput 0 # note, if you don't have spare i/o, this could negatively affect service time
 

Q3:what we need to do to reduce number of sstables per server?

Make sure you're compacting faster than you're writing and wait.

=Rob


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Robert Coli-3
On Fri, Apr 3, 2015 at 1:04 PM, Thomas Borg Salling <[hidden email]> wrote:
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.

2.1.2 had a serious bug which resulted in extra files, which is different from the overall issue I am referring to.

=Rob
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

graham sanderson
As does 2.1.3

On Apr 3, 2015, at 5:36 PM, Robert Coli <[hidden email]> wrote:

On Fri, Apr 3, 2015 at 1:04 PM, Thomas Borg Salling <[hidden email]> wrote:
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.

2.1.2 had a serious bug which resulted in extra files, which is different from the overall issue I am referring to.

=Rob
 


smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Mantas Klasavičius
Thanks a lot for all to your responses

I should mention we are running 2.1.3 and  I have set setcompactionthroughput 0 already 

nodetool enableautocompaction keyspace table command/bug is new to me I will definitely will try this out and let you know

One more thing I wan't to clarify did I understand correctly 32 is the max number for sstables for normally operating cassandra node?


Best regards
Mantas

On Sat, Apr 4, 2015 at 4:47 AM, graham sanderson <[hidden email]> wrote:
As does 2.1.3

On Apr 3, 2015, at 5:36 PM, Robert Coli <[hidden email]> wrote:

On Fri, Apr 3, 2015 at 1:04 PM, Thomas Borg Salling <[hidden email]> wrote:
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.

2.1.2 had a serious bug which resulted in extra files, which is different from the overall issue I am referring to.

=Rob
 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

graham sanderson
I have not thought thru why adding a node would cause this behavior, but


related issues (which end up with causing excessive numbers of sstables) - we saw many thousands per node for some tables

If you have a manageable number of tables, try setting coldReadsToOmit (a setting on SizeTieredCompactionStrategy) back to 0 which was the default in 2.0.x

Otherwise you could apply this patch (which reverts the default - and I doubt you had overridden it), but note that the coldReadsToOmit is fixed in 2.1.4, so if you can do it just by chaining table config then that is good.

diff --git a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactio
index fbd715c..cbb8c8b 100644
--- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
@@ -118,7 +118,11 @@ public class SizeTieredCompactionStrategy extends AbstractCompactionStrategy
     static List<SSTableReader> filterColdSSTables(List<SSTableReader> sstables, double coldReadsToOmit, int minThreshold)
     {
         if (coldReadsToOmit == 0.0)
+        {
+            if (!sstables.isEmpty())
+                logger.debug("Skipping cold sstable filter for list sized {} containing {}", sstables.size(), sstables.get(0).getFilename());
             return sstables;
+        }
 
         // Sort the sstables by hotness (coldest-first). We first build a map because the hotness may change during the sort.
         final Map<SSTableReader, Double> hotnessSnapshot = getHotnessMap(sstables);
diff --git a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java b/src/java/org/apache/cassandra/db/compaction/SizeTieredCo
index 84e7d61..c6c5f1b 100644
--- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
+++ b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
@@ -26,7 +26,7 @@ public final class SizeTieredCompactionStrategyOptions
     protected static final long DEFAULT_MIN_SSTABLE_SIZE = 50L * 1024L * 1024L;
     protected static final double DEFAULT_BUCKET_LOW = 0.5;
     protected static final double DEFAULT_BUCKET_HIGH = 1.5;
-    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.05;
+    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.0;
     protected static final String MIN_SSTABLE_SIZE_KEY = "min_sstable_size";
     protected static final String BUCKET_LOW_KEY = "bucket_low";
     protected static final String BUCKET_HIGH_KEY = "bucket_high";




On Apr 4, 2015, at 4:23 PM, Mantas Klasavičius <[hidden email]> wrote:

Thanks a lot for all to your responses

I should mention we are running 2.1.3 and  I have set setcompactionthroughput 0 already 

nodetool enableautocompaction keyspace table command/bug is new to me I will definitely will try this out and let you know

One more thing I wan't to clarify did I understand correctly 32 is the max number for sstables for normally operating cassandra node?


Best regards
Mantas

On Sat, Apr 4, 2015 at 4:47 AM, graham sanderson <[hidden email]> wrote:
As does 2.1.3

On Apr 3, 2015, at 5:36 PM, Robert Coli <[hidden email]> wrote:

On Fri, Apr 3, 2015 at 1:04 PM, Thomas Borg Salling <[hidden email]> wrote:
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.

2.1.2 had a serious bug which resulted in extra files, which is different from the overall issue I am referring to.

=Rob
 




smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Huge number of sstables after adding server to existing cluster

Anuj
We faced compaction issue with SCTS in 2.0.3. Till we upgrade, we added a dummy read every 1000 writes as workaround . Compaction started happenning in Write only heavy loads.

Anuj Wadehra
From:"graham sanderson" <[hidden email]>
Date:Sun, 5 Apr, 2015 at 9:35 am
Subject:Re: Huge number of sstables after adding server to existing cluster

I have not thought thru why adding a node would cause this behavior, but


related issues (which end up with causing excessive numbers of sstables) - we saw many thousands per node for some tables

If you have a manageable number of tables, try setting coldReadsToOmit (a setting on SizeTieredCompactionStrategy) back to 0 which was the default in 2.0.x

Otherwise you could apply this patch (which reverts the default - and I doubt you had overridden it), but note that the coldReadsToOmit is fixed in 2.1.4, so if you can do it just by chaining table config then that is good.

diff --git a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactio
index fbd715c..cbb8c8b 100644
--- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
@@ -118,7 +118,11 @@ public class SizeTieredCompactionStrategy extends AbstractCompactionStrategy
     static List<SSTableReader> filterColdSSTables(List<SSTableReader> sstables, double coldReadsToOmit, int minThreshold)
     {
         if (coldReadsToOmit == 0.0)
+        {
+            if (!sstables.isEmpty())
+                logger.debug("Skipping cold sstable filter for list sized {} containing {}", sstables.size(), sstables.get(0).getFilename());
             return sstables;
+        }
 
         // Sort the sstables by hotness (coldest-first). We first build a map because the hotness may change during the sort.
         final Map<SSTableReader, Double> hotnessSnapshot = getHotnessMap(sstables);
diff --git a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java b/src/java/org/apache/cassandra/db/compaction/SizeTieredCo
index 84e7d61..c6c5f1b 100644
--- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
+++ b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
@@ -26,7 +26,7 @@ public final class SizeTieredCompactionStrategyOptions
     protected static final long DEFAULT_MIN_SSTABLE_SIZE = 50L * 1024L * 1024L;
     protected static final double DEFAULT_BUCKET_LOW = 0.5;
     protected static final double DEFAULT_BUCKET_HIGH = 1.5;
-    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.05;
+    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.0;
     protected static final String MIN_SSTABLE_SIZE_KEY = "min_sstable_size";
     protected static final String BUCKET_LOW_KEY = "bucket_low";
     protected static final String BUCKET_HIGH_KEY = "bucket_high";




On Apr 4, 2015, at 4:23 PM, Mantas Klasavičius <<a rel="nofollow" shape="rect" class="" ymailto="mailto:mantas.klasavicius@gmail.com" target="_blank" href="javascript:return">mantas.klasavicius@...> wrote:

Thanks a lot for all to your responses

I should mention we are running 2.1.3 and  I have set setcompactionthroughput 0 already 

nodetool enableautocompaction keyspace table command/bug is new to me I will definitely will try this out and let you know

One more thing I wan't to clarify did I understand correctly 32 is the max number for sstables for normally operating cassandra node?


Best regards
Mantas

On Sat, Apr 4, 2015 at 4:47 AM, graham sanderson <<a rel="nofollow" shape="rect" class="" ymailto="mailto:graham@vast.com" target="_blank" href="javascript:return">graham@...> wrote:
As does 2.1.3

On Apr 3, 2015, at 5:36 PM, Robert Coli <<a rel="nofollow" shape="rect" class="" ymailto="mailto:rcoli@eventbrite.com" target="_blank" href="javascript:return">rcoli@...> wrote:

On Fri, Apr 3, 2015 at 1:04 PM, Thomas Borg Salling <<a rel="nofollow" shape="rect" class="" ymailto="mailto:tbsalling@tbsalling.dk" target="_blank" href="javascript:return">tbsalling@...> wrote:
I agree with Pranay. I have experienced exactly the same on C* 2.1.2.

2.1.2 had a serious bug which resulted in extra files, which is different from the overall issue I am referring to.

=Rob
 



Loading...