Bootstrap performance.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Bootstrap performance.

Dikang Gu
Hi guys,

We have a 100+ nodes cluster, each node has about 400G data, and is running on a flash disk. We are running 2.1.2.

When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network.

Does it sound normal to you?

Here are some iostat and vmstat metrics:
==== iostat ====
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          88.52    3.99    4.11    0.00    0.00    3.38

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               1.00         0.00         0.04          0          0
sdb             156.50         0.00        55.62          0        1

==== vmstat =====
138  0      0 86781912 438780 101523368    0    0     0 31893 264496 247316 95  4  1  0  0      2015-04-21 01:04:01 UTC
147  0      0 86562400 438780 101607248    0    0     0 90510 456635 245849 91  5  4  0  0      2015-04-21 01:04:03 UTC
143  0      0 86341168 438780 101692224    0    0     0 32392 284495 273656 92  4  4  0  0      2015-04-21 01:04:05 UTC

Thanks.
--
Dikang

Reply | Threaded
Open this post in threaded view
|

Re: Bootstrap performance.

Robert Coli-3
On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu <[hidden email]> wrote:
When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network.

Does it sound normal to you?

Have you unthrottled both compaction and streaming via JMX/nodetool?

Streaming is single threaded and can (?) be CPU bound, I would not be surprised if JIRA contains a ticket on the upper bounds of streaming performance in current implementation.

=Rob


 
Reply | Threaded
Open this post in threaded view
|

bootstrap performance.

Big Bear
In reply to this post by Dikang Gu
Reply | Threaded
Open this post in threaded view
|

Re: Bootstrap performance.

Dikang Gu
In reply to this post by Robert Coli-3
Hi Rob,

Why do you say steaming is single threaded? I see a lot of background streaming threads running, for example:

"STREAM-IN-/10.210.165.49" daemon prio=10 tid=0x00007f81fc001000 nid=0x107075 runnable [0x00007f836b256000]
"STREAM-IN-/10.213.51.57" daemon prio=10 tid=0x00007f81f0002000 nid=0x107073 runnable [0x00007f836b1d4000]
"STREAM-IN-/10.213.51.61" daemon prio=10 tid=0x00007f81e8001000 nid=0x107070 runnable [0x00007f836b110000]
"STREAM-IN-/10.213.51.63" daemon prio=10 tid=0x00007f81dc001800 nid=0x10706f runnable [0x00007f836b0cf000]

Thanks
Dikang.

On Mon, Apr 20, 2015 at 6:48 PM, Robert Coli <[hidden email]> wrote:
On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu <[hidden email]> wrote:
When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network.

Does it sound normal to you?

Have you unthrottled both compaction and streaming via JMX/nodetool?

Streaming is single threaded and can (?) be CPU bound, I would not be surprised if JIRA contains a ticket on the upper bounds of streaming performance in current implementation.

=Rob


 



--
Dikang

Reply | Threaded
Open this post in threaded view
|

Re: Bootstrap performance.

Robert Coli-3
On Mon, Apr 20, 2015 at 8:09 PM, Dikang Gu <[hidden email]> wrote:
Why do you say steaming is single threaded? I see a lot of background streaming threads running, for example:

Imprecise :

"Each stream is a single thread".

As I said, first place to look is throttles... but I would not be surprised if the overall number of threads available to streaming is a meaningful bound.

=Rob