Configuring multiple nodes

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuring multiple nodes

Chris Were
Hi,

I am having issues getting two cassandra instances to communicate.

 - ListenAddress on each machine to their respective public IP addresses.
 - Seeds has an entry for both public IP addresses.
 - I have opened up iptables for ports 9160, 7000, 7001, 8888 (not sure which ones really need to be opened, so did every port I could see)

Running nodeprobe on 192.168.0.101, looking at it's server ie:
$ cassandra/bin/nodeprobe -host 192.168.0.101 ring
Only shows 192.168.0.101

Running nodeprobe on 192.168.0.101, looking at the second server ie:
$ cassandra/bin/nodeprobe -host 192.168.0.102 ring
Results in the following error

Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 192.168.0.102; nested exception is: 
java.net.NoRouteToHostException: No route to host]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 192.168.0.102; nested exception is: 
java.net.NoRouteToHostException: No route to host]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:409)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
... 4 more
Caused by: java.rmi.ConnectIOException: Exception creating connection to: 192.168.0.102; nested exception is: 
java.net.NoRouteToHostException: No route to host
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:632)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:542)
at java.net.Socket.connect(Socket.java:492)
at java.net.Socket.<init>(Socket.java:389)
at java.net.Socket.<init>(Socket.java:203)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 14 more


What have I missed?

Cheers,
Chris
Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Eric Bowman
You can ping 192.168.0.102 from 192.168.0.101, right?

Assuming you can ping, do you see anything different with iptables
turned off?

cheers,
Eric

Chris Were wrote:

> Hi,
>
> I am having issues getting two cassandra instances to communicate.
>
>  - ListenAddress on each machine to their respective public IP addresses.
>  - Seeds has an entry for both public IP addresses.
>  - I have opened up iptables for ports 9160, 7000, 7001, 8888 (not
> sure which ones really need to be opened, so did every port I could see)
>
> Running nodeprobe on 192.168.0.101, looking at it's server ie:
> $ cassandra/bin/nodeprobe -host 192.168.0.101 ring
> Only shows 192.168.0.101
>
> Running nodeprobe on 192.168.0.101, looking at the second server ie:
> $ cassandra/bin/nodeprobe -host 192.168.0.102 ring
> Results in the following error
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.CommunicationException [Root exception is
> java.rmi.ConnectIOException: Exception creating connection to:
> 192.168.0.102; nested exception is:
> java.net.NoRouteToHostException: No route to host]
> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
> at
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.CommunicationException [Root exception is
> java.rmi.ConnectIOException: Exception creating connection
> to: 192.168.0.102; nested exception is:
> java.net.NoRouteToHostException: No route to host]
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
> at
> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
> at javax.naming.InitialContext.lookup(InitialContext.java:409)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
> ... 4 more
> Caused by: java.rmi.ConnectIOException: Exception creating connection
> to: 192.168.0.102; nested exception is:
> java.net.NoRouteToHostException: No route to host
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:632)
> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
> ... 9 more
> Caused by: java.net.NoRouteToHostException: No route to host
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
> at java.net.Socket.connect(Socket.java:542)
> at java.net.Socket.connect(Socket.java:492)
> at java.net.Socket.<init>(Socket.java:389)
> at java.net.Socket.<init>(Socket.java:203)
> at
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
> at
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
> ... 14 more
>
>
> What have I missed?
>
> Cheers,
> Chris


--
Eric Bowman
Boboco Ltd
[hidden email]
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532

Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Chris Were

 
Assuming you can ping, do you see anything different with iptables
turned off?

*sigh* I should have tried turning iptables off -- I can now see both nodes. Thanks :)

That being said, I'm no expert in iptables rules. This is what I added for cassandra, what rules do others use?

-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7000 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7001 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8888 -j ACCEPT

Cheers,
Chris
Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Eric Evans-4
On Sun, 2009-09-20 at 09:42 +0930, Chris Were wrote:

> *sigh* I should have tried turning iptables off -- I can now see both
> nodes. Thanks :)
>
>
> That being said, I'm no expert in iptables rules. This is what I added
> for cassandra, what rules do others use?
>
>
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9160
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7000
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7001
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8888
> -j ACCEPT

Assuming that you are using defaults, and you want to use nodeprobe (or
any other jmx client) remotely then you'll also need to open tcp/8080.

--
Eric Evans
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Anthony Molinaro-5
One other gotcha I've come across trying to use jmx was on machines with
multiple interfaces.  The rmi server doesn't always bind to the one
you want.  I add this to my JVM_OPTS in cassandra.in.sh

-Djava.rmi.server.hostname=$IP

where IP is

IP=`/sbin/ifconfig  | grep 'inet addr:' | grep -v '127.0.0.1' | cut -d: -f2 |
awk '{ print $1}'`

This cause the rmi server to bind to the appropriate address for the vmware
I run this in, allowing me to connect via jconsole running on the host OS.

I use the same thing in EC2 but the ip address is gotten from a tool which
queries the interfaces setup by the ami and finds the right one.

-Anthony

On Sun, Sep 20, 2009 at 09:09:29PM -0500, Eric Evans wrote:

> On Sun, 2009-09-20 at 09:42 +0930, Chris Were wrote:
>
> > *sigh* I should have tried turning iptables off -- I can now see both
> > nodes. Thanks :)
> >
> >
> > That being said, I'm no expert in iptables rules. This is what I added
> > for cassandra, what rules do others use?
> >
> >
> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9160
> > -j ACCEPT
> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7000
> > -j ACCEPT
> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7001
> > -j ACCEPT
> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8888
> > -j ACCEPT
>
> Assuming that you are using defaults, and you want to use nodeprobe (or
> any other jmx client) remotely then you'll also need to open tcp/8080.
>
> --
> Eric Evans
> [hidden email]
>

--
------------------------------------------------------------------------
Anthony Molinaro                           <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Jonathan Ellis-3
Could you mention this on http://wiki.apache.org/cassandra/RunningCassandra ?

On Sun, Sep 20, 2009 at 10:46 PM, Anthony Molinaro
<[hidden email]> wrote:

> One other gotcha I've come across trying to use jmx was on machines with
> multiple interfaces.  The rmi server doesn't always bind to the one
> you want.  I add this to my JVM_OPTS in cassandra.in.sh
>
> -Djava.rmi.server.hostname=$IP
>
> where IP is
>
> IP=`/sbin/ifconfig  | grep 'inet addr:' | grep -v '127.0.0.1' | cut -d: -f2 |
> awk '{ print $1}'`
>
> This cause the rmi server to bind to the appropriate address for the vmware
> I run this in, allowing me to connect via jconsole running on the host OS.
>
> I use the same thing in EC2 but the ip address is gotten from a tool which
> queries the interfaces setup by the ami and finds the right one.
>
> -Anthony
>
> On Sun, Sep 20, 2009 at 09:09:29PM -0500, Eric Evans wrote:
>> On Sun, 2009-09-20 at 09:42 +0930, Chris Were wrote:
>>
>> > *sigh* I should have tried turning iptables off -- I can now see both
>> > nodes. Thanks :)
>> >
>> >
>> > That being said, I'm no expert in iptables rules. This is what I added
>> > for cassandra, what rules do others use?
>> >
>> >
>> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9160
>> > -j ACCEPT
>> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7000
>> > -j ACCEPT
>> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7001
>> > -j ACCEPT
>> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8888
>> > -j ACCEPT
>>
>> Assuming that you are using defaults, and you want to use nodeprobe (or
>> any other jmx client) remotely then you'll also need to open tcp/8080.
>>
>> --
>> Eric Evans
>> [hidden email]
>>
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <[hidden email]>
>
Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Eric Evans-4
In reply to this post by Anthony Molinaro-5
On Sun, 2009-09-20 at 20:46 -0700, Anthony Molinaro wrote:
> One other gotcha I've come across trying to use jmx was on machines
> with multiple interfaces.  The rmi server doesn't always bind to the
> one you want.  I add this to my JVM_OPTS in cassandra.in.sh

Interesting. It always binds to all interfaces for me. If that's not
what others see, than I wonder what determines that behavior.

--
Eric Evans
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Anthony Molinaro-5

On Mon, Sep 21, 2009 at 10:08:41AM -0500, Eric Evans wrote:
> On Sun, 2009-09-20 at 20:46 -0700, Anthony Molinaro wrote:
> > One other gotcha I've come across trying to use jmx was on machines
> > with multiple interfaces.  The rmi server doesn't always bind to the
> > one you want.  I add this to my JVM_OPTS in cassandra.in.sh
>
> Interesting. It always binds to all interfaces for me. If that's not
> what others see, than I wonder what determines that behavior.

I think by default it uses the ip address of "localhost" which is fine
if you have a local jconsole.

http://download.java.net/jdk7/docs/technotes/guides/rmi/javarmiproperties.html

I tend to always use a remote jconsole so the rmi server needs to be bound
to the correct interface.  I saw something mentioning that rmi hardcodes
the ip in the stubs it generates and hands out.

-Anthony

--
------------------------------------------------------------------------
Anthony Molinaro                           <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Configuring multiple nodes

Anthony Molinaro-5
In reply to this post by Jonathan Ellis-3
Okay, added a couple of sentences about this.

On Sun, Sep 20, 2009 at 10:52:49PM -0500, Jonathan Ellis wrote:

> Could you mention this on http://wiki.apache.org/cassandra/RunningCassandra ?
>
> On Sun, Sep 20, 2009 at 10:46 PM, Anthony Molinaro
> <[hidden email]> wrote:
> > One other gotcha I've come across trying to use jmx was on machines with
> > multiple interfaces.  The rmi server doesn't always bind to the one
> > you want.  I add this to my JVM_OPTS in cassandra.in.sh
> >
> > -Djava.rmi.server.hostname=$IP
> >
> > where IP is
> >
> > IP=`/sbin/ifconfig  | grep 'inet addr:' | grep -v '127.0.0.1' | cut -d: -f2 |
> > awk '{ print $1}'`
> >
> > This cause the rmi server to bind to the appropriate address for the vmware
> > I run this in, allowing me to connect via jconsole running on the host OS.
> >
> > I use the same thing in EC2 but the ip address is gotten from a tool which
> > queries the interfaces setup by the ami and finds the right one.
> >
> > -Anthony
> >
> > On Sun, Sep 20, 2009 at 09:09:29PM -0500, Eric Evans wrote:
> >> On Sun, 2009-09-20 at 09:42 +0930, Chris Were wrote:
> >>
> >> > *sigh* I should have tried turning iptables off -- I can now see both
> >> > nodes. Thanks :)
> >> >
> >> >
> >> > That being said, I'm no expert in iptables rules. This is what I added
> >> > for cassandra, what rules do others use?
> >> >
> >> >
> >> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9160
> >> > -j ACCEPT
> >> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7000
> >> > -j ACCEPT
> >> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 7001
> >> > -j ACCEPT
> >> > -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8888
> >> > -j ACCEPT
> >>
> >> Assuming that you are using defaults, and you want to use nodeprobe (or
> >> any other jmx client) remotely then you'll also need to open tcp/8080.
> >>
> >> --
> >> Eric Evans
> >> [hidden email]
> >>
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro                           <[hidden email]>
> >

--
------------------------------------------------------------------------
Anthony Molinaro                           <[hidden email]>