Column value not getting updated

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Column value not getting updated

Saurabh Sethi
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh
Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

Nate McCall-4
You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

Saurabh Sethi
That’s what I found out that the clocks were not in sync.

But I have setup NTP on all 3 nodes and would expect the clocks to be in sync.

From: Nate McCall <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 2:50 PM
To: Cassandra Users <[hidden email]>
Subject: Re: Column value not getting updated

You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

Jonathan Haddad
It's not enough to set up ntp, you're going to need to force the time to sync.  ntp is only meant to correct for drift.

You can either use ntpdate or I think there's a flag for ntpd (that I can't remember and am in a rush out the door) that you can use to force it to adjust to the correct time.

On Tue, Mar 31, 2015 at 3:02 PM Saurabh Sethi <[hidden email]> wrote:
That’s what I found out that the clocks were not in sync.

But I have setup NTP on all 3 nodes and would expect the clocks to be in sync.

From: Nate McCall <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 2:50 PM
To: Cassandra Users <[hidden email]>
Subject: Re: Column value not getting updated

You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

Mark Greene
In reply to this post by Saurabh Sethi
Hey Saurabh,

We're actually preparing for this ourselves and spinning up our own NTP server pool. The public NTP pools have a lot of drift and should not be relied upon for cluster technology that is sensitive to time skew like C*.

The folks at Logentries did a great write up about this which we used as a guide.

-Mark

On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi <[hidden email]> wrote:
That’s what I found out that the clocks were not in sync.

But I have setup NTP on all 3 nodes and would expect the clocks to be in sync.

From: Nate McCall <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 2:50 PM
To: Cassandra Users <[hidden email]>
Subject: Re: Column value not getting updated

You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

Saurabh Sethi
Thanks Mark. A great post indeed and saved me a lot of trouble.

- Saurabh
From: Mark Greene <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 10:15 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Column value not getting updated

Hey Saurabh,

We're actually preparing for this ourselves and spinning up our own NTP server pool. The public NTP pools have a lot of drift and should not be relied upon for cluster technology that is sensitive to time skew like C*.

The folks at Logentries did a great write up about this which we used as a guide.

-Mark

On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi <[hidden email]> wrote:
That’s what I found out that the clocks were not in sync.

But I have setup NTP on all 3 nodes and would expect the clocks to be in sync.

From: Nate McCall <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 2:50 PM
To: Cassandra Users <[hidden email]>
Subject: Re: Column value not getting updated

You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Reply | Threaded
Open this post in threaded view
|

Re: Column value not getting updated

daemeon reiydelle
Interesting that you are finding excessive drift from public time servers. I only once saw that problem with AWS' time servers. To be conservative I sometimes recommend that clients spool up their own time server, but realize IT will also drift if the public time servers do! Somewhat different if in your own DC, but same time server drift issues.

Google has resorted to putting tier one time server(s) (cesium clock or whatever) in every data center due to the public drift issues. Does anyone know if AWS' time server is now stratum 1 backed?

However, it is better to have two (at least) in AWS, make sure their private IP's are not in the same 24 CIDR subnet!

Of course this can get troublesome if load sharing between e.g. AWS East and West.


.......
“Life should not be a journey to the grave with the intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of smoke,
thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!”

- Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872


On Tue, Mar 31, 2015 at 10:49 PM, Saurabh Sethi <[hidden email]> wrote:
Thanks Mark. A great post indeed and saved me a lot of trouble.

- Saurabh
From: Mark Greene <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 10:15 PM
To: "[hidden email]" <[hidden email]>

Subject: Re: Column value not getting updated

Hey Saurabh,

We're actually preparing for this ourselves and spinning up our own NTP server pool. The public NTP pools have a lot of drift and should not be relied upon for cluster technology that is sensitive to time skew like C*.

The folks at Logentries did a great write up about this which we used as a guide.

-Mark

On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi <[hidden email]> wrote:
That’s what I found out that the clocks were not in sync.

But I have setup NTP on all 3 nodes and would expect the clocks to be in sync.

From: Nate McCall <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, March 31, 2015 at 2:50 PM
To: Cassandra Users <[hidden email]>
Subject: Re: Column value not getting updated

You would see that if the servers' clocks were out of sync. 

Make sure the time on the servers is in sync or set the client timestamps explicitly. 

On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi <[hidden email]> wrote:
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns.

After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value.
  • I am using QueryBuilder API and not CQL directly.
  • I am using the consistency level of QUORUM for everything – insert, update and read.
  • Cassandra is running as a 3 node cluster with replication factor of 3.

Anyone has any idea what is going on here?

Thanks,
Saurabh



--
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com