Connecting to Cassandra cluster in AWS from local network

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Connecting to Cassandra cluster in AWS from local network

Matthew Johnson

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 

Reply | Threaded
Open this post in threaded view
|

Re: Connecting to Cassandra cluster in AWS from local network

Jonathan Haddad
Ideally you'll be on the same network, but if you can't be, you'll need to use the public ip in listen_address.

On Mon, Apr 20, 2015 at 9:47 AM Matthew Johnson <[hidden email]> wrote:

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 

Reply | Threaded
Open this post in threaded view
|

Re: Connecting to Cassandra cluster in AWS from local network

Russell Bradberry
There are a couple options here. You can use the built in address translator, or, write a new load balancing policy.  See https://datastax-oss.atlassian.net/browse/JAVA-145 for more information.

From: Jonathan Haddad
Reply-To: <[hidden email]>
Date: Monday, April 20, 2015 at 12:50 PM
To: <[hidden email]>
Subject: Re: Connecting to Cassandra cluster in AWS from local network

Ideally you'll be on the same network, but if you can't be, you'll need to use the public ip in listen_address.

On Mon, Apr 20, 2015 at 9:47 AM Matthew Johnson <[hidden email]> wrote:

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 

Reply | Threaded
Open this post in threaded view
|

Re: Connecting to Cassandra cluster in AWS from local network

Alex Popescu-2
In reply to this post by Jonathan Haddad
You'll have to configure your nodes to:

1. use AWS internal IPs for inter-node connection (check listen_address) and 
2. use the AWS public IP for client-to-node connections (check rpc_address)

Depending on the setup, there might be other interesting conf options in cassandra.yaml (broadcast_address, listen_interface, rpc_interface).


On Mon, Apr 20, 2015 at 9:50 AM, Jonathan Haddad <[hidden email]> wrote:
Ideally you'll be on the same network, but if you can't be, you'll need to use the public ip in listen_address.

On Mon, Apr 20, 2015 at 9:47 AM Matthew Johnson <[hidden email]> wrote:

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 




--
Bests,

Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax

Reply | Threaded
Open this post in threaded view
|

Re: Connecting to Cassandra cluster in AWS from local network

Russell Bradberry
I would like to note that this will require all clients connect over the external IP address. If you have clients within Amazon that need to connect over the private IP address, this would not be possible.  If you have a mix of clients that need to connect over private IP address and public, then one of the solutions outlined in https://datastax-oss.atlassian.net/browse/JAVA-145 may be more appropriate.

-Russ

From: Alex Popescu
Reply-To: <[hidden email]>
Date: Monday, April 20, 2015 at 2:00 PM
To: user
Subject: Re: Connecting to Cassandra cluster in AWS from local network

You'll have to configure your nodes to:

1. use AWS internal IPs for inter-node connection (check listen_address) and 
2. use the AWS public IP for client-to-node connections (check rpc_address)

Depending on the setup, there might be other interesting conf options in cassandra.yaml (broadcast_address, listen_interface, rpc_interface).


On Mon, Apr 20, 2015 at 9:50 AM, Jonathan Haddad <[hidden email]> wrote:
Ideally you'll be on the same network, but if you can't be, you'll need to use the public ip in listen_address.

On Mon, Apr 20, 2015 at 9:47 AM Matthew Johnson <[hidden email]> wrote:

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 




--
Bests,

Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax

Reply | Threaded
Open this post in threaded view
|

RE: Connecting to Cassandra cluster in AWS from local network

Matthew Johnson

Thanks everyone for the suggestions!

 

I have used the following code to create my cluster from my dev environment and it seems to be working perfectly:

 

            cluster = Cluster.builder().addContactPoints(nodes).withAddressTranslater(new AddressTranslater() {

                  public InetSocketAddress translate(InetSocketAddress address) {

                        String newAddress = null;

                        if(address != null && address.getAddress() != null) {

                              if(address.getHostName().equals("172.x.x.237")) newAddress = "54.x.x.157";

                              if(address.getHostName().equals("172.x.x.170")) newAddress = "54.x.x.208";

                              if(address.getHostName().equals("172.x.x.150")) newAddress = "54.x.x.142";

                        }

                        return new InetSocketAddress(newAddress, address.getPort());

                  }

            }).build();

 

 

Cheers,

Matt

 

 

From: Russell Bradberry [mailto:[hidden email]]
Sent: 20 April 2015 19:06
To: [hidden email]
Subject: Re: Connecting to Cassandra cluster in AWS from local network

 

I would like to note that this will require all clients connect over the external IP address. If you have clients within Amazon that need to connect over the private IP address, this would not be possible.  If you have a mix of clients that need to connect over private IP address and public, then one of the solutions outlined in https://datastax-oss.atlassian.net/browse/JAVA-145 may be more appropriate.

 

-Russ

 

From: Alex Popescu
Reply-To: <[hidden email]>
Date: Monday, April 20, 2015 at 2:00 PM
To: user
Subject: Re: Connecting to Cassandra cluster in AWS from local network

 

You'll have to configure your nodes to:

 

1. use AWS internal IPs for inter-node connection (check listen_address) and 

2. use the AWS public IP for client-to-node connections (check rpc_address)

 

Depending on the setup, there might be other interesting conf options in cassandra.yaml (broadcast_address, listen_interface, rpc_interface).

 

 

On Mon, Apr 20, 2015 at 9:50 AM, Jonathan Haddad <[hidden email]> wrote:

Ideally you'll be on the same network, but if you can't be, you'll need to use the public ip in listen_address.

 

On Mon, Apr 20, 2015 at 9:47 AM Matthew Johnson <[hidden email]> wrote:

Hi all,

 

I have set up a Cassandra cluster with 2.1.4 on some existing AWS boxes, just as a POC. Cassandra servers connect to each other over their internal AWS IP addresses (172.x.x.x) aliased in /etc/hosts as sales1, sales2 and sales3.

 

I connect to it from my local dev environment using the seed’s external NAT address (54.x.x.x) aliases in my Windows hosts file as sales3 (my seed).

 

When I try to connect, it connects fine, and can retrieve some data (I have very limited amounts of data in there, but it seems to retrieve ok), but I also get lots of stacktraces in my log where my dev environment is trying to connect to Cassandra on the internal IP (presumably the Cassandra seed node tells my dev env where to look):

 

 

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host sales3/54.x.x.142:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.237:9042 added

INFO  2015-04-20 16:34:14,808 [CASSANDRA-CLIENT] {main} Cluster - New Cassandra host /172.x.x.170:9042 added

Connected to cluster: Test Cluster

Datatacenter: datacenter1; Host: /172.x.x.170; Rack: rack1

Datatacenter: datacenter1; Host: sales3/54.x.x.142; Rack: rack1

Datatacenter: datacenter1; Host: /172.x.x.237; Rack: rack1

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Connection - Connection[sales3/54.x.x.142:9042-2, inFlight=0, closed=false] Transport initialized and ready

DEBUG 2015-04-20 16:34:14,901 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-0} Session - Added connection pool for sales3/54.x.x.142:9042

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Connection[/172.x.x.237:9042-1, inFlight=0, closed=false] Error connecting to /172.x.x.237:9042 (connection timed out: /172.x.x.237:9042)

DEBUG 2015-04-20 16:34:19,850 [CASSANDRA-CLIENT] {Cassandra Java Driver worker-1} Connection - Defuncting connection to /172.x.x.237:9042

com.datastax.driver.core.TransportException: [/172.x.x.237:9042] Cannot connect

 

 

Does anyone have any experience with connecting to AWS clusters from dev machines? How have you set up your aliases to get around this issue?

 

Current setup in sales3 (seed node) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales3

rpc_address: sales3

 

Current setup in other nodes (eg sales2) cassandra.yaml:

 

- seeds: "sales3"

listen_address: sales2

rpc_address: sales2

 

 

Thanks!

Matt

 



 

--

Bests,


Alex Popescu | @al3xandru

Sen. Product Manager @ DataStax