write timeout

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

write timeout

Anishek Agarwal
Hello,

I am using a single node  server class machine with 16 CPUs with 32GB RAM with a single drive attached to it.

my table structure is as below
CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
I am trying to insert 300 entries per partition key with 4000 partition keys using 25 threads. Configurations

write_request_timeout_in_ms: 5000
concurrent_writes: 32
heap space : 8GB

Client side timeout is 12 sec using datastax java driver.
Consistency level: ONE

With the above configuration i try to run it 10 times to eventually generate around

300 * 4000 * 10 = 12000000 entries,

When i run this after the first few runs i get a WriteTimeout exception at client with "1 replica were required but only 0 acknowledged the write" message.

There are no errors in server log. Why does this error come how do i know what is the limit I should limit concurrent writes to a single node to.


Looking at iostat disk utilization seems to be at 1-3% when running this.

Please let me know if anything else is required.

Regards,
Anishek

Reply | Threaded
Open this post in threaded view
|

Re: write timeout

Anishek Agarwal
Forgot to mention I am using Cassandra 2.0.13

On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal <[hidden email]> wrote:
Hello,

I am using a single node  server class machine with 16 CPUs with 32GB RAM with a single drive attached to it.

my table structure is as below
CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
I am trying to insert 300 entries per partition key with 4000 partition keys using 25 threads. Configurations

write_request_timeout_in_ms: 5000
concurrent_writes: 32
heap space : 8GB

Client side timeout is 12 sec using datastax java driver.
Consistency level: ONE

With the above configuration i try to run it 10 times to eventually generate around

300 * 4000 * 10 = 12000000 entries,

When i run this after the first few runs i get a WriteTimeout exception at client with "1 replica were required but only 0 acknowledged the write" message.

There are no errors in server log. Why does this error come how do i know what is the limit I should limit concurrent writes to a single node to.


Looking at iostat disk utilization seems to be at 1-3% when running this.

Please let me know if anything else is required.

Regards,
Anishek


Reply | Threaded
Open this post in threaded view
|

Re: write timeout

Brian Tarbox-2
My group is seeing the same thing and also can not figure out why its happening.

On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <[hidden email]> wrote:
Forgot to mention I am using Cassandra 2.0.13

On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal <[hidden email]> wrote:
Hello,

I am using a single node  server class machine with 16 CPUs with 32GB RAM with a single drive attached to it.

my table structure is as below
CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
I am trying to insert 300 entries per partition key with 4000 partition keys using 25 threads. Configurations

write_request_timeout_in_ms: 5000
concurrent_writes: 32
heap space : 8GB

Client side timeout is 12 sec using datastax java driver.
Consistency level: ONE

With the above configuration i try to run it 10 times to eventually generate around

300 * 4000 * 10 = 12000000 entries,

When i run this after the first few runs i get a WriteTimeout exception at client with "1 replica were required but only 0 acknowledged the write" message.

There are no errors in server log. Why does this error come how do i know what is the limit I should limit concurrent writes to a single node to.


Looking at iostat disk utilization seems to be at 1-3% when running this.

Please let me know if anything else is required.

Regards,
Anishek





--
Reply | Threaded
Open this post in threaded view
|

Re: write timeout

Robert Coli-3
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <[hidden email]> wrote:
My group is seeing the same thing and also can not figure out why its happening.

On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <[hidden email]> wrote:
Forgot to mention I am using Cassandra 2.0.13

This seems like a rather significant bug in the most recent stable version. In this case, I would tend to file a JIRA first and then ask the mailing list second.

Could one or both of you file steps-to-reproduce with a JIRA at http://issues.apache.org?

=Rob
 
Reply | Threaded
Open this post in threaded view
|

Re: write timeout

Anishek Agarwal

On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <[hidden email]> wrote:
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <[hidden email]> wrote:

> My group is seeing the same thing and also can not figure out why its
> happening.
>
> On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <[hidden email]>
> wrote:
>
>> Forgot to mention I am using Cassandra 2.0.13
>>
>
This seems like a rather significant bug in the most recent stable version.
In this case, I would tend to file a JIRA first and then ask the mailing
list second.

Could one or both of you file steps-to-reproduce with a JIRA at
http://issues.apache.org?

=Rob

Reply | Threaded
Open this post in threaded view
|

Re: write timeout

Anishek Agarwal
after a few more trials I was able to spot some errors in the system.log, attaching the log here for reference.

Looks like "test-t1-jb-78" is opened by SSTableReader in some batch but then later there is compaction that is happening on it where in the file is not found. Any idea what is happening?




On Tue, Mar 24, 2015 at 9:54 AM, Anishek Agarwal <[hidden email]> wrote:

On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <[hidden email]> wrote:
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <[hidden email]> wrote:

> My group is seeing the same thing and also can not figure out why its
> happening.
>
> On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <[hidden email]>
> wrote:
>
>> Forgot to mention I am using Cassandra 2.0.13
>>
>
This seems like a rather significant bug in the most recent stable version.
In this case, I would tend to file a JIRA first and then ask the mailing
list second.

Could one or both of you file steps-to-reproduce with a JIRA at
http://issues.apache.org?

=Rob



out_system.log (500K) Download Attachment