rpc_timeout error

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

rpc_timeout error

Amlan Roy
Hi,

I am trying to query a table from cqlsh and I get the following error:
Request did not complete within rpc_timeout.

I found the following messages in system.log:
system.log:ERROR [ReadStage:1493] 2015-04-12 00:46:39,754 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in my_keyspace.my_event_data; query aborted (see tombstone_failure_threshold)

 I  thought the tombstones are created only during delete. I am not doing any update/delete on this table, just doing inserts. I have not set any TTL also. Not sure what is the issue here.

Regards,
Amlan
Reply | Threaded
Open this post in threaded view
|

Re: rpc_timeout error

Jonathan Haddad
Maybe you're inserting nulls?  If you're inserting nulls, those show up as tombstones.  

On Sat, Apr 11, 2015 at 1:32 PM Amlan Roy <[hidden email]> wrote:
Hi,

I am trying to query a table from cqlsh and I get the following error:
Request did not complete within rpc_timeout.

I found the following messages in system.log:
system.log:ERROR [ReadStage:1493] 2015-04-12 00:46:39,754 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in my_keyspace.my_event_data; query aborted (see tombstone_failure_threshold)

 I  thought the tombstones are created only during delete. I am not doing any update/delete on this table, just doing inserts. I have not set any TTL also. Not sure what is the issue here.

Regards,
Amlan
Reply | Threaded
Open this post in threaded view
|

Re: rpc_timeout error

Amlan Roy
Hi,

Thanks for the reply. Yes, I am inserting nulls as some columns are unknown for some rows. Will this issue get resolved if I start inserting empty strings instead of null? Can it be because I have a very wide row which I am updating throghout the day (basically inserting time series data)? This is the output of the “node tool cfstats” command for the table:

SSTable count: 8
Space used (live), bytes: 12813800373
Space used (total), bytes: 12813800373
Off heap memory used (total), bytes: 9268093
SSTable Compression Ratio: 0.1664112140412863
Number of keys (estimate): 1024
Memtable cell count: 1143127
Memtable data size, bytes: 248893920
Memtable switch count: 551
Local read count: 43
Local read latency: 0.000 ms
Local write count: 12211615
Local write latency: 2.451 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used, bytes: 6576
Bloom filter off heap memory used, bytes: 6512
Index summary off heap memory used, bytes: 357
Compression metadata off heap memory used, bytes: 9261224
Compacted partition minimum bytes: 4769
Compacted partition maximum bytes: 2395318855
Compacted partition mean bytes: 382807869
Average live cells per slice (last five minutes): 9081.0
Average tombstones per slice (last five minutes): 40169.0

Regards.
On 12-Apr-2015, at 2:38 am, Jonathan Haddad <[hidden email]> wrote:

Maybe you're inserting nulls?  If you're inserting nulls, those show up as tombstones.  

On Sat, Apr 11, 2015 at 1:32 PM Amlan Roy <[hidden email]> wrote:
Hi,

I am trying to query a table from cqlsh and I get the following error:
Request did not complete within rpc_timeout.

I found the following messages in system.log:
system.log:ERROR [ReadStage:1493] 2015-04-12 00:46:39,754 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in my_keyspace.my_event_data; query aborted (see tombstone_failure_threshold)

 I  thought the tombstones are created only during delete. I am not doing any update/delete on this table, just doing inserts. I have not set any TTL also. Not sure what is the issue here.

Regards,
Amlan

Reply | Threaded
Open this post in threaded view
|

Re: rpc_timeout error

Jonathan Haddad
Can't you just not insert anything?  

On Sat, Apr 11, 2015 at 2:24 PM Amlan Roy <[hidden email]> wrote:
Hi,

Thanks for the reply. Yes, I am inserting nulls as some columns are unknown for some rows. Will this issue get resolved if I start inserting empty strings instead of null? Can it be because I have a very wide row which I am updating throghout the day (basically inserting time series data)? This is the output of the “node tool cfstats” command for the table:

SSTable count: 8
Space used (live), bytes: 12813800373
Space used (total), bytes: 12813800373
Off heap memory used (total), bytes: 9268093
SSTable Compression Ratio: 0.1664112140412863
Number of keys (estimate): 1024
Memtable cell count: 1143127
Memtable data size, bytes: 248893920
Memtable switch count: 551
Local read count: 43
Local read latency: 0.000 ms
Local write count: 12211615
Local write latency: 2.451 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used, bytes: 6576
Bloom filter off heap memory used, bytes: 6512
Index summary off heap memory used, bytes: 357
Compression metadata off heap memory used, bytes: 9261224
Compacted partition minimum bytes: 4769
Compacted partition maximum bytes: 2395318855
Compacted partition mean bytes: 382807869
Average live cells per slice (last five minutes): 9081.0
Average tombstones per slice (last five minutes): 40169.0

Regards.
On 12-Apr-2015, at 2:38 am, Jonathan Haddad <[hidden email]> wrote:

Maybe you're inserting nulls?  If you're inserting nulls, those show up as tombstones.  

On Sat, Apr 11, 2015 at 1:32 PM Amlan Roy <[hidden email]> wrote:
Hi,

I am trying to query a table from cqlsh and I get the following error:
Request did not complete within rpc_timeout.

I found the following messages in system.log:
system.log:ERROR [ReadStage:1493] 2015-04-12 00:46:39,754 SliceQueryFilter.java (line 206) Scanned over 100000 tombstones in my_keyspace.my_event_data; query aborted (see tombstone_failure_threshold)

 I  thought the tombstones are created only during delete. I am not doing any update/delete on this table, just doing inserts. I have not set any TTL also. Not sure what is the issue here.

Regards,
Amlan