FileNotFoundException

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

FileNotFoundException

bbatranut
Hello all,

One of my C* throws a big amount of exceptions like this:


ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:792,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:87)
        at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1439)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2004)
        ... 3 more
Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
        ... 29 more


I must say, that this node was started with cleared data / commit log and it was joining an existing cluster. I see a lot of these exceptions. Not sure what to do.

Reply | Threaded
Open this post in threaded view
|

Fw: FileNotFoundException

bbatranut
Also I must add that grepping the logs for a particular file I see this:

 INFO [CompactionExecutor:19] 2015-02-24 10:44:35,618 CompactionTask.java (line 120) Compacting [SSTableReader(path='/data/ranks/positions/ranks-positions-jb-339-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-354-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-408-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-286-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-20-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-127-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-357-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-257-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-316-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-41-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-285-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-338-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-180-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-398-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-249-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-284-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-294-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-248-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-377-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-395-Data.d ...

also several entries like this in the log after grep.
java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)
Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)



I was grepping for jb-41-Data.db ...  seems that this file does not exist for some reason. I must say that when I first added the node I included it's IP in the seeds list. Then I have decommissioned it, removed it's IP from the seed list, deleted all data / commit log / saved caches and started it. Since then I have not manualy deleted any files . 

Any ideeas?


On Tuesday, February 24, 2015 11:46 AM, Batranut Bogdan <[hidden email]> wrote:


Hello all,

One of my C* throws a big amount of exceptions like this:


ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:792,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:87)
        at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1439)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2004)
        ... 3 more
Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
        ... 29 more


I must say, that this node was started with cleared data / commit log and it was joining an existing cluster. I see a lot of these exceptions. Not sure what to do.



Jan
Reply | Threaded
Open this post in threaded view
|

Re: FileNotFoundException

Jan
HI Batranut;

In both errors you described above the files seem to be missing while compaction is running. 
Without knowing what else is going on your system,  I would presume that this error occurs on this single node only and not your entire cluster. 

Some guesses:
a)  You may have a disk corruption problem.     Take the node offline and run a diskcheck.
b)   Take the node offline, wipe it clean of everything and have it rejoin the cluster.   Check if the problem recurs.

Hope this helps
Jan





On Tuesday, February 24, 2015 2:56 AM, Batranut Bogdan <[hidden email]> wrote:


Also I must add that grepping the logs for a particular file I see this:

 INFO [CompactionExecutor:19] 2015-02-24 10:44:35,618 CompactionTask.java (line 120) Compacting [SSTableReader(path='/data/ranks/positions/ranks-positions-jb-339-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-354-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-408-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-286-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-20-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-127-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-357-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-257-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-316-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-41-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-285-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-338-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-180-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-398-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-249-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-284-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-294-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-248-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-377-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-395-Data.d ...

also several entries like this in the log after grep.
java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)
Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)



I was grepping for jb-41-Data.db ...  seems that this file does not exist for some reason. I must say that when I first added the node I included it's IP in the seeds list. Then I have decommissioned it, removed it's IP from the seed list, deleted all data / commit log / saved caches and started it. Since then I have not manualy deleted any files . 

Any ideeas?


On Tuesday, February 24, 2015 11:46 AM, Batranut Bogdan <[hidden email]> wrote:


Hello all,

One of my C* throws a big amount of exceptions like this:


ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:792,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:87)
        at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1439)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2004)
        ... 3 more
Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
        ... 29 more


I must say, that this node was started with cleared data / commit log and it was joining an existing cluster. I see a lot of these exceptions. Not sure what to do.