Keyspace Replication changes not synchronized after adding Datacenter

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Keyspace Replication changes not synchronized after adding Datacenter

Thunder Stumpges
Hi guys,

We have recently added two datacenters to our existing 2.0.6 cluster. We followed the process here pretty much exactly:
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

We are using GossipingPropertyFileSnitch and NetworkTopologyStrategy across the board. All property files are identical in each of the three datacenters, and we use two nodes from each DC in the seed list.

However when we came to step 7.a. we ran the ALTER KEYSPACE command on one of the new datacenters (to add it as a replica). This change was reflected on the new datacenter where it ran as returned by DESCRIBE KEYSPACE. However the change was NOT propagated to either of the other two datacenters. We effectively had to run the ALTER KEYSPACE command 3 times, one in each datacenter. Is this expected? I could find no documentation stating that this needed to be done, nor any documentation around how the system keyspace was kept in sync across datacenters in general.

If this is indicative of a larger problem with our installation, how would we go about troubleshooting it?

Thanks in advance!
Thunder


Reply | Threaded
Open this post in threaded view
|

Re: Keyspace Replication changes not synchronized after adding Datacenter

Paul Leddy
Hello,

No, that is not expected at all, to run the alter statement in each DC. Yes, it indicates a larger problem, for sure.

Check that ports are open between all nodes, especially 7000, if I recall correctly. We use a simple telnet check.

Paul

On 04/13/2015 10:22 AM, Thunder Stumpges wrote:
Hi guys,

We have recently added two datacenters to our existing 2.0.6 cluster. We followed the process here pretty much exactly:
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

We are using GossipingPropertyFileSnitch and NetworkTopologyStrategy across the board. All property files are identical in each of the three datacenters, and we use two nodes from each DC in the seed list.

However when we came to step 7.a. we ran the ALTER KEYSPACE command on one of the new datacenters (to add it as a replica). This change was reflected on the new datacenter where it ran as returned by DESCRIBE KEYSPACE. However the change was NOT propagated to either of the other two datacenters. We effectively had to run the ALTER KEYSPACE command 3 times, one in each datacenter. Is this expected? I could find no documentation stating that this needed to be done, nor any documentation around how the system keyspace was kept in sync across datacenters in general.

If this is indicative of a larger problem with our installation, how would we go about troubleshooting it?

Thanks in advance!
Thunder