Cassandra cluster Too high DISK IOs

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Cassandra cluster Too high DISK IOs

Pranay Agarwal
Hi All.


I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS disks (4000). I can see around 12k reads/sec ops on the cassandra cluster. 

But I see around ~3500 read IOPS on each of the cassandra nodes. Is that normal?

I am using LevelledCompaction and I can see in the histograms that most read requests are coming from 1/2 sstables only. Why are there so many disk IOPS or is the normal for cassandra?

-Pranay
Reply | Threaded
Open this post in threaded view
|

Re: Cassandra cluster Too high DISK IOs

Ali Akhtar
That probably depends on how many read / write queries your cluster is processing?

Also, since you mentioned provisoned IOPS, are you using EBS for storing the data? If so, you probably want to switch to the ephemeral storage since its locally attached to the instance and doesn't require a network call for each lookup.

On Fri, Mar 20, 2015 at 10:57 PM, Pranay Agarwal <[hidden email]> wrote:
Hi All.


I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS disks (4000). I can see around 12k reads/sec ops on the cassandra cluster. 

But I see around ~3500 read IOPS on each of the cassandra nodes. Is that normal?

I am using LevelledCompaction and I can see in the histograms that most read requests are coming from 1/2 sstables only. Why are there so many disk IOPS or is the normal for cassandra?

-Pranay

Reply | Threaded
Open this post in threaded view
|

Re: Cassandra cluster Too high DISK IOs

Pranay Agarwal
The cluster is processing something like 12k reads and 2k writes/seconds. The disks are locally attached and latency is just fine. It's the number of disk iops that's too high.

On Fri, Mar 20, 2015 at 11:05 AM, Ali Akhtar <[hidden email]> wrote:
That probably depends on how many read / write queries your cluster is processing?

Also, since you mentioned provisoned IOPS, are you using EBS for storing the data? If so, you probably want to switch to the ephemeral storage since its locally attached to the instance and doesn't require a network call for each lookup.

On Fri, Mar 20, 2015 at 10:57 PM, Pranay Agarwal <[hidden email]> wrote:
Hi All.


I am using 15 nodes cassandra cluster(m3.2xlarge) with provisioned IOPS disks (4000). I can see around 12k reads/sec ops on the cassandra cluster. 

But I see around ~3500 read IOPS on each of the cassandra nodes. Is that normal?

I am using LevelledCompaction and I can see in the histograms that most read requests are coming from 1/2 sstables only. Why are there so many disk IOPS or is the normal for cassandra?

-Pranay


Reply | Threaded
Open this post in threaded view
|

Re: Cassandra cluster Too high DISK IOs

Duncan Sands
On 20/03/15 19:34, Pranay Agarwal wrote:
> The cluster is processing something like 12k reads and 2k writes/seconds. The
> disks are locally attached and latency is just fine. It's the number of disk
> iops that's too high.

Maybe each read is accessing many sstables.

Ciao, Duncan.
Reply | Threaded
Open this post in threaded view
|

Re: Cassandra cluster Too high DISK IOs

Pranay Agarwal
No. as shown in the histograms, 99% of reads are using 2 or less number of tables. What's typical usually? Can anyone share from experience?

On Fri, Mar 20, 2015 at 1:12 PM, Duncan Sands <[hidden email]> wrote:
On 20/03/15 19:34, Pranay Agarwal wrote:
The cluster is processing something like 12k reads and 2k writes/seconds. The
disks are locally attached and latency is just fine. It's the number of disk
iops that's too high.

Maybe each read is accessing many sstables.

Ciao, Duncan.