Why so many SSTables?
Locked
Why so many SSTables?
 
|
Hi, We are surprised by the number of files generated by Cassandra. Our cluster consists of 9 nodes and each node handles about 35 GB. We're using Cassandra 1.0.6 with LeveledCompactionStrategy. We have 30 CF. We've got roughly 45,000 files under the keyspace directory on each node: ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l 44372 The biggest CF is spread over 38,000 files: ls -l Documents* | wc -l 37870 ls -l Documents*-Data.db | wc -l 7586 Many SSTable are about 4 MB: 19 MB -> 1 SSTable 12 MB -> 2 SSTables 11 MB -> 2 SSTables 9.2 MB -> 1 SSTable 7.0 MB to 7.9 MB -> 6 SSTables 6.0 MB to 6.4 MB -> 6 SSTables 5.0 MB to 5.4 MB -> 4 SSTables 4.0 MB to 4.7 MB -> 7139 SSTables 3.0 MB to 3.9 MB -> 258 SSTables 2.0 MB to 2.9 MB -> 35 SSTables 1.0 MB to 1.9 MB -> 13 SSTables 87 KB to 994 KB -> 87 SSTables 0 KB -> 32 SSTables FYI here is CF information: ColumnFamily: Documents Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds / keys to save : 0.0/0/all Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider Key cache size / save period in seconds: 200000.0/14400 GC grace seconds: 1728000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Column Metadata: Column Name: refUUID (72656655554944) Validation Class: org.apache.cassandra.db.marshal.BytesType Index Name: refUUID_idx Index Type: KEYS Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor Is it a bug? If not, how can we tune Cassandra to avoid this? Regards, Romain |
Re: Why so many SSTables?
|
LCS explicitly tries to keep sstables under 5MB to minimize extra work
done by compacting data that didn't really overlap across different levels. On Tue, Apr 10, 2012 at 9:24 AM, Romain HARDOUIN <[hidden email]> wrote: > > Hi, > > We are surprised by the number of files generated by Cassandra. > Our cluster consists of 9 nodes and each node handles about 35 GB. > We're using Cassandra 1.0.6 with LeveledCompactionStrategy. > We have 30 CF. > > We've got roughly 45,000 files under the keyspace directory on each node: > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l > 44372 > > The biggest CF is spread over 38,000 files: > ls -l Documents* | wc -l > 37870 > > ls -l Documents*-Data.db | wc -l > 7586 > > Many SSTable are about 4 MB: > > 19 MB -> 1 SSTable > 12 MB -> 2 SSTables > 11 MB -> 2 SSTables > 9.2 MB -> 1 SSTable > 7.0 MB to 7.9 MB -> 6 SSTables > 6.0 MB to 6.4 MB -> 6 SSTables > 5.0 MB to 5.4 MB -> 4 SSTables > 4.0 MB to 4.7 MB -> 7139 SSTables > 3.0 MB to 3.9 MB -> 258 SSTables > 2.0 MB to 2.9 MB -> 35 SSTables > 1.0 MB to 1.9 MB -> 13 SSTables > 87 KB to 994 KB -> 87 SSTables > 0 KB -> 32 SSTables > > FYI here is CF information: > > ColumnFamily: Documents > Key Validation Class: org.apache.cassandra.db.marshal.BytesType > Default column value validator: org.apache.cassandra.db.marshal.BytesType > Columns sorted by: org.apache.cassandra.db.marshal.BytesType > Row cache size / save period in seconds / keys to save : 0.0/0/all > Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider > Key cache size / save period in seconds: 200000.0/14400 > GC grace seconds: 1728000 > Compaction min/max thresholds: 4/32 > Read repair chance: 1.0 > Replicate on write: true > Column Metadata: > Column Name: refUUID (72656655554944) > Validation Class: org.apache.cassandra.db.marshal.BytesType > Index Name: refUUID_idx > Index Type: KEYS > Compaction Strategy: > org.apache.cassandra.db.compaction.LeveledCompactionStrategy > Compression Options: > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor > > Is it a bug? If not, how can we tune Cassandra to avoid this? > > Regards, > > Romain -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com |
Re: Why so many SSTables?
|
In reply to this post by Romain Hardouin
You can configure sstable size by sstable_size_in_mb parameter for LCS.
The default value is 5MB. You should better to check you don't have many pending compaction tasks with nodetool tpstats and compactionstats also. If you have enough IO throughput, you can increase compaction_throughput_mb_per_sec in cassandra.yaml to reduce pending compactions. maki 2012/4/10 Romain HARDOUIN <[hidden email]>: > > Hi, > > We are surprised by the number of files generated by Cassandra. > Our cluster consists of 9 nodes and each node handles about 35 GB. > We're using Cassandra 1.0.6 with LeveledCompactionStrategy. > We have 30 CF. > > We've got roughly 45,000 files under the keyspace directory on each node: > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l > 44372 > > The biggest CF is spread over 38,000 files: > ls -l Documents* | wc -l > 37870 > > ls -l Documents*-Data.db | wc -l > 7586 > > Many SSTable are about 4 MB: > > 19 MB -> 1 SSTable > 12 MB -> 2 SSTables > 11 MB -> 2 SSTables > 9.2 MB -> 1 SSTable > 7.0 MB to 7.9 MB -> 6 SSTables > 6.0 MB to 6.4 MB -> 6 SSTables > 5.0 MB to 5.4 MB -> 4 SSTables > 4.0 MB to 4.7 MB -> 7139 SSTables > 3.0 MB to 3.9 MB -> 258 SSTables > 2.0 MB to 2.9 MB -> 35 SSTables > 1.0 MB to 1.9 MB -> 13 SSTables > 87 KB to 994 KB -> 87 SSTables > 0 KB -> 32 SSTables > > FYI here is CF information: > > ColumnFamily: Documents > Key Validation Class: org.apache.cassandra.db.marshal.BytesType > Default column value validator: org.apache.cassandra.db.marshal.BytesType > Columns sorted by: org.apache.cassandra.db.marshal.BytesType > Row cache size / save period in seconds / keys to save : 0.0/0/all > Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider > Key cache size / save period in seconds: 200000.0/14400 > GC grace seconds: 1728000 > Compaction min/max thresholds: 4/32 > Read repair chance: 1.0 > Replicate on write: true > Column Metadata: > Column Name: refUUID (72656655554944) > Validation Class: org.apache.cassandra.db.marshal.BytesType > Index Name: refUUID_idx > Index Type: KEYS > Compaction Strategy: > org.apache.cassandra.db.compaction.LeveledCompactionStrategy > Compression Options: > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor > > Is it a bug? If not, how can we tune Cassandra to avoid this? > > Regards, > > Romain |
Re: Why so many SSTables?
|
|
Thank you for your answers. I originally post this question because we encoutered an OOM Exception on 2 nodes during repair session. Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner which contains as many objects there are SSTables on disk (7747 objects at the time). This ArrayList consumes 47% of the heap space (786 MB). We want each node to handle 1 TB, so we must dramatically reduce the number of SSTables. Thus, is there any drawback if we set sstable_size_in_mb to 200MB? Otherwise shoudl we go back to Tiered Compaction? Regards, Romain Maki Watanabe <[hidden email]> a écrit sur 11/04/2012 04:21:47 : > You can configure sstable size by sstable_size_in_mb parameter for LCS. > The default value is 5MB. > You should better to check you don't have many pending compaction tasks > with nodetool tpstats and compactionstats also. > If you have enough IO throughput, you can increase > compaction_throughput_mb_per_sec > in cassandra.yaml to reduce pending compactions. > > maki > > 2012/4/10 Romain HARDOUIN <[hidden email]>: > > > > Hi, > > > > We are surprised by the number of files generated by Cassandra. > > Our cluster consists of 9 nodes and each node handles about 35 GB. > > We're using Cassandra 1.0.6 with LeveledCompactionStrategy. > > We have 30 CF. > > > > We've got roughly 45,000 files under the keyspace directory on each node: > > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l > > 44372 > > > > The biggest CF is spread over 38,000 files: > > ls -l Documents* | wc -l > > 37870 > > > > ls -l Documents*-Data.db | wc -l > > 7586 > > > > Many SSTable are about 4 MB: > > > > 19 MB -> 1 SSTable > > 12 MB -> 2 SSTables > > 11 MB -> 2 SSTables > > 9.2 MB -> 1 SSTable > > 7.0 MB to 7.9 MB -> 6 SSTables > > 6.0 MB to 6.4 MB -> 6 SSTables > > 5.0 MB to 5.4 MB -> 4 SSTables > > 4.0 MB to 4.7 MB -> 7139 SSTables > > 3.0 MB to 3.9 MB -> 258 SSTables > > 2.0 MB to 2.9 MB -> 35 SSTables > > 1.0 MB to 1.9 MB -> 13 SSTables > > 87 KB to 994 KB -> 87 SSTables > > 0 KB -> 32 SSTables > > > > FYI here is CF information: > > > > ColumnFamily: Documents > > Key Validation Class: org.apache.cassandra.db.marshal.BytesType > > Default column value validator: org.apache.cassandra.db.marshal.BytesType > > Columns sorted by: org.apache.cassandra.db.marshal.BytesType > > Row cache size / save period in seconds / keys to save : 0.0/0/all > > Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider > > Key cache size / save period in seconds: 200000.0/14400 > > GC grace seconds: 1728000 > > Compaction min/max thresholds: 4/32 > > Read repair chance: 1.0 > > Replicate on write: true > > Column Metadata: > > Column Name: refUUID (72656655554944) > > Validation Class: org.apache.cassandra.db.marshal.BytesType > > Index Name: refUUID_idx > > Index Type: KEYS > > Compaction Strategy: > > org.apache.cassandra.db.compaction.LeveledCompactionStrategy > > Compression Options: > > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor > > > > Is it a bug? If not, how can we tune Cassandra to avoid this? > > > > Regards, > > > > Romain |
Re: Why so many SSTables?
|
On Wed, Apr 11, 2012 at 2:43 PM, Romain HARDOUIN
<[hidden email]> wrote: > > Thank you for your answers. > > I originally post this question because we encoutered an OOM Exception on 2 > nodes during repair session. > Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner > which contains as many objects there are SSTables on disk (7747 objects at > the time). > This ArrayList consumes 47% of the heap space (786 MB). That's 101KB per element!! I know Java object representation is not concise but that feels like more than is reasonable. Are you sure of those numbers? In any case, we should improve that so as to not create all those SSTableBoundedScanner upfront. Would you mind opening a ticket on https://issues.apache.org/jira/browse/CASSANDRA with as many info as you have on this. > We want each node to handle 1 TB, so we must dramatically reduce the number > of SSTables. > > Thus, is there any drawback if we set sstable_size_in_mb to 200MB? > Otherwise shoudl we go back to Tiered Compaction? > > Regards, > > Romain > > > Maki Watanabe <[hidden email]> a écrit sur 11/04/2012 04:21:47 : > > >> You can configure sstable size by sstable_size_in_mb parameter for LCS. >> The default value is 5MB. >> You should better to check you don't have many pending compaction tasks >> with nodetool tpstats and compactionstats also. >> If you have enough IO throughput, you can increase >> compaction_throughput_mb_per_sec >> in cassandra.yaml to reduce pending compactions. >> >> maki >> >> 2012/4/10 Romain HARDOUIN <[hidden email]>: >> > >> > Hi, >> > >> > We are surprised by the number of files generated by Cassandra. >> > Our cluster consists of 9 nodes and each node handles about 35 GB. >> > We're using Cassandra 1.0.6 with LeveledCompactionStrategy. >> > We have 30 CF. >> > >> > We've got roughly 45,000 files under the keyspace directory on each >> > node: >> > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l >> > 44372 >> > >> > The biggest CF is spread over 38,000 files: >> > ls -l Documents* | wc -l >> > 37870 >> > >> > ls -l Documents*-Data.db | wc -l >> > 7586 >> > >> > Many SSTable are about 4 MB: >> > >> > 19 MB -> 1 SSTable >> > 12 MB -> 2 SSTables >> > 11 MB -> 2 SSTables >> > 9.2 MB -> 1 SSTable >> > 7.0 MB to 7.9 MB -> 6 SSTables >> > 6.0 MB to 6.4 MB -> 6 SSTables >> > 5.0 MB to 5.4 MB -> 4 SSTables >> > 4.0 MB to 4.7 MB -> 7139 SSTables >> > 3.0 MB to 3.9 MB -> 258 SSTables >> > 2.0 MB to 2.9 MB -> 35 SSTables >> > 1.0 MB to 1.9 MB -> 13 SSTables >> > 87 KB to 994 KB -> 87 SSTables >> > 0 KB -> 32 SSTables >> > >> > FYI here is CF information: >> > >> > ColumnFamily: Documents >> > Key Validation Class: org.apache.cassandra.db.marshal.BytesType >> > Default column value validator: >> > org.apache.cassandra.db.marshal.BytesType >> > Columns sorted by: org.apache.cassandra.db.marshal.BytesType >> > Row cache size / save period in seconds / keys to save : 0.0/0/all >> > Row Cache Provider: >> > org.apache.cassandra.cache.SerializingCacheProvider >> > Key cache size / save period in seconds: 200000.0/14400 >> > GC grace seconds: 1728000 >> > Compaction min/max thresholds: 4/32 >> > Read repair chance: 1.0 >> > Replicate on write: true >> > Column Metadata: >> > Column Name: refUUID (72656655554944) >> > Validation Class: org.apache.cassandra.db.marshal.BytesType >> > Index Name: refUUID_idx >> > Index Type: KEYS >> > Compaction Strategy: >> > org.apache.cassandra.db.compaction.LeveledCompactionStrategy >> > Compression Options: >> > sstable_compression: >> > org.apache.cassandra.io.compress.SnappyCompressor >> > >> > Is it a bug? If not, how can we tune Cassandra to avoid this? >> > >> > Regards, >> > >> > Romain |
Re: Why so many SSTables?
|
In reply to this post by Romain Hardouin
It's easy to spend other people's money, but handling 1TB of data
with 1.5 g heap? Memory is cheap, and just a little more will solve
many problems.
On 04/11/2012 08:43 AM, Romain HARDOUIN wrote:
|
Re: Why so many SSTables?
|
In general I would limit the data load per node to 300 to 400GB. Otherwise things can painful when it comes time to run compaction / repair / move .
Cheers On 12/04/2012, at 1:00 AM, Dave Brosius wrote:
|
Re: Why so many SSTables?
|
>In general I would limit the data load per node to 300 to 400GB. Otherwise > things can painful when it comes time to run compaction / repair / move . |
Re: Why so many SSTables?
|
|
In reply to this post by Romain Hardouin
If you increase sstable_size_in_mb to 200MB, you will need more IO for each compaction. For example, if your memtable will be flushed, and LCS needs to compact it with 10 overwrapped L1 sstables, you will need almost 2GB read and 2GB write for the single compaction. From iPhone
|
Re: Why so many SSTables?
|
|
I've just opened a new JIRA: CASSANDRA-4142 I've double checked numbers, 7747 seems to be array list object's capacity (Eclipse Memory Analyzer displays "java.lang.Object[7747] @ 0x7d3f3f798"). Actually there are 5757 browsable entries in EMA therefore each object is about 140 KB (size varies between 143088 bits and 143768 bits). We have no pending compactions tasks. Our cluster is currently under-loaded. Our goal is to handle hundreds of tera bytes which explains 1 TB per node. Our need is to archive data so our cluster is not running under a read-heavy load. @Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak suspects* as reported by Eclipse Memory Analyser. Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to 2,5 GB. Regards, Romain |
Re: Why so many SSTables?
|
From my experience I would strongly advise against leveled compaction
for your use-case. But you should certainly test and see for yourself! I have ~1TB on a node with ~13GB of heap. I ended up with 30k SSTables. I raised the SSTable size to 100MB but that didn't prove to be sufficient and I did it too late. I also had issues along the way where I ran out of heap space (before going to 13GB) and the flushing that then happens caused lots of small SSTables to be produced. Loading up 30k SSTables at start-up took 30+ minutes. With 30k SSTables I had 5 levels, if I remember correctly, that means everything goes through 5 compactions in the long run and the resulting cpu load is just "phenomenal". I was using snappy compressor, maybe that exacerbated the problem. At some point I overloaded thing with writes and had a compaction backlog of over 3000. I stopped all data loading and after several days it was still compacting away and not about to end. I eventually switched to size tiered compaction and life is back to normal... Of course, leveled compaction may work wonderfully for others, this is just what I experienced. If I were to try it again, I'd watch my SSTable count and abort the experiment if I end up with >10k and I'd watch my compaction backlog and abort if it goes above 100. - Thorsten On 4/12/2012 12:54 AM, Romain HARDOUIN wrote: > > I've just opened a new JIRA: CASSANDRA-4142 > > I've double checked numbers, 7747 seems to be array list object's > capacity (Eclipse Memory Analyzer displays "java.lang.Object[7747] @ > 0x7d3f3f798"). > Actually there are 5757 browsable entries in EMA therefore each object > is about 140 KB (size varies between 143088 bits and 143768 bits). > > We have no pending compactions tasks. Our cluster is currently > under-loaded. > > Our goal is to handle hundreds of tera bytes which explains 1 TB per > node. Our need is to archive data so our cluster is not running under > a read-heavy load. > > @Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak > suspects* as reported by Eclipse Memory Analyser. > Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to > 2,5 GB. > > Regards, > > Romain |
RMI/JMX errors, weird
|
Hello,
I'm doing compactions under 0.8.8. Recently, I started seeing a stack trace like one below, and I can't figure out what causes this to appear. The cluster has been in operation for mode than half a year w/o errors like this one. Any help will be appreciated, Thanks Maxim WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out Exception in thread "main" java.io.IOException: Repair command #1: some repair session(s) failed (see log for details). at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) |
Re: RMI/JMX errors, weird
|
|
Look at the server side logs for errors.
Cheers On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
|
Re: RMI/JMX errors, weird
|
|
Server log below. Mind you that all the nodes are still up -- even
though reported as "dead" in this log.
What's going on here? Thanks! INFO [GossipTasks:1] 2012-04-18 22:18:26,487 Gossiper.java (line 719) InetAddress /130.199.185.193 is now dead. INFO [ScheduledTasks:1] 2012-04-18 22:18:26,487 StatusLogger.java (line 50) Pool Name Active Pending Blocked ERROR [GossipTasks:1] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 722) Problem during repair session manual-repair-1b3453b 6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,488 StorageService.java (line 1607) Repair session org.apache.cas sandra.service.AntiEntropyService$RepairSession@4cc9e2bc failed. java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai r-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.185.195 died at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509b bfec9872, endpoint /130.199.185.195 died at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199. 185.195 died at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723) at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760) at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165) at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538) at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57) at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) ... 3 more INFO [FlushWriter:4] 2012-04-18 22:18:26,489 Memtable.java (line 237) Writing Memtable-files.pandaid@988533693(1754325/140346000 se rialized/live bytes, 70173 ops) ERROR [AntiEntropySessions:8] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 691) Repair session manual-repair-1b3453b6-28b5- 4abd-84ce-0326b5468064 (on cfs [Ljava.lang.String;@5a44cc06, range (113427455640312821154458202477256070484,0]) failed with the foll owing error java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 die d at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723) at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760) at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165) at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538) at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57) at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) INFO [ScheduledTasks:1] 2012-04-18 22:18:26,488 StatusLogger.java (line 65) ReadStage 0 0 0 ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,491 StorageService.java (line 1607) Repair session org.apache.cas sandra.service.AntiEntropyService$RepairSession@41bb342c failed. java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai r-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326 b5468064, endpoint /130.199.185.193 died at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199. 185.193 died at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723) at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760) at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165) at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538) at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57) at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) On 4/12/2012 10:03 PM, aaron morton wrote: Look at the server side logs for errors. |
Re: RMI/JMX errors, weird
|
|
At some point the gossip system on the node this log is from decided that 130.199.185.195 was DOWN. This was based on how often the node was gossiping to the cluster.
The active repair session was informed. And to avoid failing the job unnecessarily it tested that the errant nodes phi value was twice the configured phi_convict_threshold. It was and the repair was killed.
Take a look at the logs on 130.199.185.195 and see if anything was happening on the node at the same time. Could be GC or an overloaded node (it would log about dropped messages). Perhaps other nodes also saw 130.199.185.195 as down? it only needed to be down for a few seconds. Hope that helps. On 19/04/2012, at 2:25 PM, Maxim Potekhin wrote:
|
Re: RMI/JMX errors, weird
|
|
Hello Aaron,
how should I go about fixing that? Also, after a repeated attempt to compact it goes again into "building secondary index" with 1150 pending tasks, and is not making progress. I suspected the disk system failure, but this needs to be confirmed. So basically, do I need to tune the phi threshold up? Thing is, there was no heavy load on the cluster at all. Thanks Maxim On 4/19/2012 7:06 AM, aaron morton wrote: At some point the gossip system on the node this log is from decided that 130.199.185.195 was DOWN. This was based on how often the node was gossiping to the cluster. |
Re: RMI/JMX errors, weird
|
|
Not all pending tasks reported by nodetool compactionstats actually run. Once they get a chance to run the files they were going to work on may have already been compacted. Did other nodes notice it was dead ? Was there anything in the log that showed it was under duress (GC or dropped message logs) ? Is the compaction a consequence of repair ? (The streaming stage can result in compactions). Or do you think the node is just behind on compactions ? If you feel compaction is hurting the node, consider setting concurrent_compactors in the yaml to 2. You can also isolate the node from updates using nodetool disablegossip and disablerthrift , and the turn off the IO limiter with nodetool setcompactionthroughput 0. Hope that helps. On 20/04/2012, at 12:29 AM, Maxim Potekhin wrote:
|
Re: RMI/JMX errors, weird
|
|
Hello Aaron,
it's probably the over-optimistic number of concurrent compactors that was tripping the system. I do not entirely understand what's the correlation here, maybe it's that the compactors were overloading the neighboring nodes causing time-outs. I tuned the concurrency down and after a while things seem to have settled down, thanks for the suggestion. Maxim On 4/19/2012 4:13 PM, aaron morton wrote:
|
«
Return to cassandra-user@incubator.apache.org
|
1 view|%1 views
Loading...
| Powered by Nabble | Edit this page |