tuning for read performance

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

tuning for read performance

feedly team
Hi,
    I have a small 2 node cassandra cluster that seems to be constrained by read throughput. There are about 100 writes/s and 60 reads/s mostly against a skinny column family. Here's the cfstats for that family:

 SSTable count: 13
 Space used (live): 231920026568
 Space used (total): 231920026568
 Number of Keys (estimate): 356899200
 Memtable Columns Count: 1385568
 Memtable Data Size: 359155691
 Memtable Switch Count: 26
 Read Count: 40705879
 Read Latency: 25.010 ms.
 Write Count: 9680958
 Write Latency: 0.036 ms.
 Pending Tasks: 0
 Bloom Filter False Postives: 28380
 Bloom Filter False Ratio: 0.00360
 Bloom Filter Space Used: 874173664
 Compacted row minimum size: 61
 Compacted row maximum size: 152321
 Compacted row mean size: 1445

iostat shows almost no write activity, here's a typical line:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               0.00     0.00  312.87    0.00     6.61     0.00    43.27    23.35  105.06   2.28  71.19

and nodetool tpstats always shows pending tasks in the ReadStage. The data set has grown beyond physical memory (250GB/node w/64GB of RAM) so I know disk access is required, but are there particular settings I should experiment with that could help relieve some read i/o pressure? I already put memcached in front of cassandra so the row cache probably won't help much. 

Also this column family stores smallish documents (usually 1-100K) along with metadata. The document is only occasionally accessed, usually only the metadata is read/written. Would splitting out the document into a separate column family help?

Thanks
Kireet
Reply | Threaded
Open this post in threaded view
|

Re: tuning for read performance

Aaron Turner
On Mon, Oct 22, 2012 at 11:05 AM, feedly team <[hidden email]> wrote:

> Hi,
>     I have a small 2 node cassandra cluster that seems to be constrained by
> read throughput. There are about 100 writes/s and 60 reads/s mostly against
> a skinny column family. Here's the cfstats for that family:
>
>  SSTable count: 13
>  Space used (live): 231920026568
>  Space used (total): 231920026568
>  Number of Keys (estimate): 356899200
>  Memtable Columns Count: 1385568
>  Memtable Data Size: 359155691
>  Memtable Switch Count: 26
>  Read Count: 40705879
>  Read Latency: 25.010 ms.
>  Write Count: 9680958
>  Write Latency: 0.036 ms.
>  Pending Tasks: 0
>  Bloom Filter False Postives: 28380
>  Bloom Filter False Ratio: 0.00360
>  Bloom Filter Space Used: 874173664
>  Compacted row minimum size: 61
>  Compacted row maximum size: 152321
>  Compacted row mean size: 1445
>
> iostat shows almost no write activity, here's a typical line:
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sdb               0.00     0.00  312.87    0.00     6.61     0.00    43.27
> 23.35  105.06   2.28  71.19
>
> and nodetool tpstats always shows pending tasks in the ReadStage. The data
> set has grown beyond physical memory (250GB/node w/64GB of RAM) so I know
> disk access is required, but are there particular settings I should
> experiment with that could help relieve some read i/o pressure? I already
> put memcached in front of cassandra so the row cache probably won't help
> much.
>
> Also this column family stores smallish documents (usually 1-100K) along
> with metadata. The document is only occasionally accessed, usually only the
> metadata is read/written. Would splitting out the document into a separate
> column family help?
>

Some un-expert advice:

1. Consider Leveled compaction instead of Size Tiered.  LCS improves
read performance at the cost of more writes.

2. You said "skinny column family" which I took to mean not a lot of
columns/row.  See if you can organize your data into wider rows which
allow reading fewer rows and thus fewer queries/disk seeks.

3. Enable compression if you haven't already.

4. Splitting your data from your MetaData could definitely help.  I
like separating my read heavy from write heavy CF's because generally
speaking they benefit from different compaction methods.  But don't go
crazy creating 1000's of CF's either.

Hope that gives you some ideas to investigate further!


--
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin
"carpe diem quam minimum credula postero"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for read performance

aaron morton
and nodetool tpstats always shows pending tasks in the ReadStage.
Are clients reading a single row at a time or multiple rows ? Each row requested in a multi get becomes a task in the read stage. 

Also look at the type of query you are sending. I talked a little about the performance of different query techniques at Cassandra SFhttp://www.datastax.com/events/cassandrasummit2012/presentations

 
1. Consider Leveled compaction instead of Size Tiered.  LCS improves
read performance at the cost of more writes.
I would look at other options first. 
If you want to know how many SSTables a read is hitting look at nodetool cfhistograms

2. You said "skinny column family" which I took to mean not a lot of
columns/row.  See if you can organize your data into wider rows which
allow reading fewer rows and thus fewer queries/disk seeks.
Wide rows take longer to read than narrow ones. Artificially wide rows may take longer to read than narrow ones. 


4. Splitting your data from your MetaData could definitely help.  I
like separating my read heavy from write heavy CF's because generally
speaking they benefit from different compaction methods.  But don't go
crazy creating 1000's of CF's either.
+1
25 ms read latency is high. 

Hope that helps. 

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 23/10/2012, at 9:06 AM, Aaron Turner <[hidden email]> wrote:

On Mon, Oct 22, 2012 at 11:05 AM, feedly team <[hidden email]> wrote:
Hi,
   I have a small 2 node cassandra cluster that seems to be constrained by
read throughput. There are about 100 writes/s and 60 reads/s mostly against
a skinny column family. Here's the cfstats for that family:

SSTable count: 13
Space used (live): 231920026568
Space used (total): 231920026568
Number of Keys (estimate): 356899200
Memtable Columns Count: 1385568
Memtable Data Size: 359155691
Memtable Switch Count: 26
Read Count: 40705879
Read Latency: 25.010 ms.
Write Count: 9680958
Write Latency: 0.036 ms.
Pending Tasks: 0
Bloom Filter False Postives: 28380
Bloom Filter False Ratio: 0.00360
Bloom Filter Space Used: 874173664
Compacted row minimum size: 61
Compacted row maximum size: 152321
Compacted row mean size: 1445

iostat shows almost no write activity, here's a typical line:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await  svctm  %util
sdb               0.00     0.00  312.87    0.00     6.61     0.00    43.27
23.35  105.06   2.28  71.19

and nodetool tpstats always shows pending tasks in the ReadStage. The data
set has grown beyond physical memory (250GB/node w/64GB of RAM) so I know
disk access is required, but are there particular settings I should
experiment with that could help relieve some read i/o pressure? I already
put memcached in front of cassandra so the row cache probably won't help
much.

Also this column family stores smallish documents (usually 1-100K) along
with metadata. The document is only occasionally accessed, usually only the
metadata is read/written. Would splitting out the document into a separate
column family help?


Some un-expert advice:

1. Consider Leveled compaction instead of Size Tiered.  LCS improves
read performance at the cost of more writes.

2. You said "skinny column family" which I took to mean not a lot of
columns/row.  See if you can organize your data into wider rows which
allow reading fewer rows and thus fewer queries/disk seeks.

3. Enable compression if you haven't already.

4. Splitting your data from your MetaData could definitely help.  I
like separating my read heavy from write heavy CF's because generally
speaking they benefit from different compaction methods.  But don't go
crazy creating 1000's of CF's either.

Hope that gives you some ideas to investigate further!


--
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
   -- Benjamin Franklin
"carpe diem quam minimum credula postero"