nodetool cleanup error

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

nodetool cleanup error

Amlan Roy
Hi,

I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I am getting the following error. Wanted to know if there is any solution to it.

Regards,
Amlan

Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Duncan Sands
Hi Amlan,

On 30/03/15 22:12, Amlan Roy wrote:

> Hi,
>
> I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I
> am getting the following error. Wanted to know if there is any solution to it.
>
> Regards,
> Amlan
>
> Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed

this is fixed in 2.0.13.

Best wishes, Duncan.
Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Jeff Ferland
In reply to this post by Amlan Roy
Code problem that was patched in https://issues.apache.org/jira/browse/CASSANDRA-8716. Upgrade to 2.0.13


On Mar 30, 2015, at 1:12 PM, Amlan Roy <[hidden email]> wrote:

Hi,

I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I am getting the following error. Wanted to know if there is any solution to it.

Regards,
Amlan

Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)

Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Amlan Roy
In reply to this post by Duncan Sands
Hi,

Thanks for the reply. I have upgraded to 2.0.13. Now I get the following error.

Regards,
Amlan

Exception in thread "main" java.lang.AssertionError: [SSTableReader(path='/data/1/cassandra/data/xxx/xxx/xxx.db'), SSTableReader(path='/data/1/cassandra/data/xxx/xxx/xxx.db')]
at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2176)
at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2173)
at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2155)
at org.apache.cassandra.db.ColumnFamilyStore.markAllCompacting(ColumnFamilyStore.java:2186)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:215)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1117)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

On 31-Mar-2015, at 1:50 am, Duncan Sands <[hidden email]> wrote:

Hi Amlan,

On 30/03/15 22:12, Amlan Roy wrote:
Hi,

I have added new nodes to an existing cluster and ran the “nodetool cleanup”. I
am getting the following error. Wanted to know if there is any solution to it.

Regards,
Amlan

Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed

this is fixed in 2.0.13.

Best wishes, Duncan.

Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Robert Coli-3
On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <[hidden email]> wrote:
Thanks for the reply. I have upgraded to 2.0.13. Now I get the following error.

If cleanup is still excepting for you on 2.0.13 with some sstables you have, I would strongly consider :

1) file a JIRA (http://issues.apache.org) and attach / offer the sstables for debugging
2) let the list know the JIRA id of the ticket

=Rob

Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Yuki Morishita
Looks like the issue is https://issues.apache.org/jira/browse/CASSANDRA-9070.

On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <[hidden email]> wrote:

> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <[hidden email]> wrote:
>>
>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
>> error.
>
>
> If cleanup is still excepting for you on 2.0.13 with some sstables you have,
> I would strongly consider :
>
> 1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
> for debugging
> 2) let the list know the JIRA id of the ticket
>
> =Rob
>



--
Yuki Morishita
 t:yukim (http://twitter.com/yukim)
Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Amlan Roy
Hi,

Thanks for the reply. Since nodetool cleanup is not working even after upgrading to 2.0.13, is it recommended to go to an older version (2.0.11 for example, with 2.0.12 also it did not work). Is there any other way of cleaning data from existing nodes after adding a new node.

Regards,
Amlan

On 31-Mar-2015, at 5:00 am, Yuki Morishita <[hidden email]> wrote:

> Looks like the issue is https://issues.apache.org/jira/browse/CASSANDRA-9070.
>
> On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <[hidden email]> wrote:
>> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <[hidden email]> wrote:
>>>
>>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
>>> error.
>>
>>
>> If cleanup is still excepting for you on 2.0.13 with some sstables you have,
>> I would strongly consider :
>>
>> 1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
>> for debugging
>> 2) let the list know the JIRA id of the ticket
>>
>> =Rob
>>
>
>
>
> --
> Yuki Morishita
> t:yukim (http://twitter.com/yukim)

Reply | Threaded
Open this post in threaded view
|

Re: nodetool cleanup error

Marcus Eriksson
It should work on 2.0.13. If it fails with that assertion, you should just retry. If that does not work, and you can reproduce this, please file a ticket

/Marcus

On Tue, Mar 31, 2015 at 9:33 AM, Amlan Roy <[hidden email]> wrote:
Hi,

Thanks for the reply. Since nodetool cleanup is not working even after upgrading to 2.0.13, is it recommended to go to an older version (2.0.11 for example, with 2.0.12 also it did not work). Is there any other way of cleaning data from existing nodes after adding a new node.

Regards,
Amlan

On 31-Mar-2015, at 5:00 am, Yuki Morishita <[hidden email]> wrote:

> Looks like the issue is https://issues.apache.org/jira/browse/CASSANDRA-9070.
>
> On Mon, Mar 30, 2015 at 6:25 PM, Robert Coli <[hidden email]> wrote:
>> On Mon, Mar 30, 2015 at 4:21 PM, Amlan Roy <[hidden email]> wrote:
>>>
>>> Thanks for the reply. I have upgraded to 2.0.13. Now I get the following
>>> error.
>>
>>
>> If cleanup is still excepting for you on 2.0.13 with some sstables you have,
>> I would strongly consider :
>>
>> 1) file a JIRA (http://issues.apache.org) and attach / offer the sstables
>> for debugging
>> 2) let the list know the JIRA id of the ticket
>>
>> =Rob
>>
>
>
>
> --
> Yuki Morishita
> t:yukim (http://twitter.com/yukim)