DSE 4.6 with OpsCenter 5.1.1, agent can't start, port 9042 is occupied by DSE

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

DSE 4.6 with OpsCenter 5.1.1, agent can't start, port 9042 is occupied by DSE

Serega Sheypak
Hi, getting weird problem when agent to connect to OpsCenter
OpsCenter installed on VM with DSE and agent.
It's not for production, I have 3 VMs with DSE and OpsCenter for dev/test purposes.

The stacktrace from agent log is:

vagrant@dsenode03:~$ sudo cat /var/log/datastax-agent/agent.log

 Starting DataStax agent monitor datastax_agent_monitor

 INFO [main] 2015-04-05 13:32:31,594 Loading conf files: /var/lib/datastax-agent/conf/address.yaml

 INFO [main] 2015-04-05 13:32:31,642 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_76

 INFO [main] 2015-04-05 13:32:31,642 DataStax Agent version: 5.1.1

 INFO [main] 2015-04-05 13:32:31,679 Default config values: {:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings", :restore_req_update_period 60, :my_channel_prefix "/agent", :poll_period 60, :thrift_conn_timeout 10000, :rollups60_ttl 604800, :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :max-seconds-to-sleep 25, :private-conf-props ["initial_token" "listen_address" "broadcast_address" "rpc_address"], :thrift_port 9160, :async_retry_timeout 5, :agent-conf-group "global-cluster-agent-group", :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1, :async_queue_size 5000, :backup_staging_dir nil, :read-buffer-size 10000000, :remote_verify_max 300000, :disk_usage_update_period 60, :throttle-bytes-per-second 500000, :rollups7200_ttl 31536000, :remote_backup_retries 3, :ssl_keystore nil, :rollup_snapshot_period 300, :is_package true, :monitor_command "/usr/share/datastax-agent/bin/datastax_agent_monitor", :thrift_socket_timeout 5000, :remote_verify_initial_delay 1000, :cassandra_log_location "/var/log/cassandra/system.log", :remote_backup_region "us-west-1", :restore_on_transfer_failure false, :tmp_dir "/var/lib/datastax-agent/tmp/", :config_md5 nil, :jmx_port 7199, :write-buffer-size 100000, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl 0, :nodedetails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :backup_file_queue_max 10000, :jmx_thread_pool_size 5, :production 1, :runs_sudo 1, :max_file_transfer_attempts 30, :stomp_interface nil, :storage_keyspace "OpsCenter", :hosts ["127.0.0.1"], :rollup_snapshot_threshold 300, :jmx_retry_timeout 30, :unthrottled-default 10000000000, :remote_backup_retry_delay 5000, :remote_backup_timeout 1000, :seconds-to-read-kill-channel 0.005, :realtime_interval 5, :pdps_ttl 259200}

 INFO [main] 2015-04-05 13:32:31,924 Waiting for the config from OpsCenter

 INFO [main] 2015-04-05 13:32:31,925 Attempting to determine Cassandra's broadcast address through JMX

 INFO [Initialization] 2015-04-05 13:32:31,926 New JMX connection (127.0.0.1:7199)

 INFO [main] 2015-04-05 13:32:31,947 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}

 INFO [Jetty] 2015-04-05 13:32:32,026 Jetty server started

 INFO [Initialization] 2015-04-05 13:32:32,054 Using 192.168.56.30 as the cassandra broadcast address

 INFO [Initialization] 2015-04-05 13:32:32,135 cassandra RPC address is  nil

 INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC address is  192.168.56.30

 INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC broadcast address is  192.168.56.30

ERROR [Initialization] 2015-04-05 13:32:32,342 Can't connect to Cassandra, retrying

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot connect))

at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)

at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)

at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)

at com.datastax.driver.core.Cluster.init(Cluster.java:158)

at com.datastax.driver.core.Cluster.connect(Cluster.java:246)

at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)

at clojure.lang.RestFn.invoke(RestFn.java:410)

at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)

at opsagent.cassandra$setup_cassandra$fn__2491.invoke(cassandra.clj:269)

at again.core$with_retries_STAR_$fn__2363.invoke(core.clj:98)

at again.core$with_retries_STAR_.invoke(core.clj:97)

at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:267)

at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:152)

at opsagent.jmx$determine_ip.invoke(jmx.clj:276)

at opsagent.jmx$setup_jmx$fn__2867.invoke(jmx.clj:293)

at clojure.lang.AFn.run(AFn.java:24)

at java.lang.Thread.run(Thread.java:745)


And it tries to repeat in several times.


I did:

vagrant@dsenode03:~$ sudo netstat -alnpt | grep 9042

tcp        0      0 192.168.56.30:9042      0.0.0.0:*               LISTEN      5490/java       


and see that some stuff already listening to this port

# cut some output

vagrant@dsenode03:~$ sudo lsof -p 5490

COMMAND  PID      USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME

java    5490 cassandra  cwd    DIR              252,0     4096    3670233 /usr/share/dse

java    5490 cassandra  rtd    DIR              252,0     4096          2 /


# cut some output

vagrant@dsenode03:~$ sudo lsof -p 5490 | grep .log     

java    5490 cassandra  mem    REG              252,0 33554432    3803832 /var/lib/cassandra/commitlog/CommitLog-3-1428238370080.log

java    5490 cassandra  mem    REG              252,0 33554432    3803355 /var/lib/cassandra/commitlog/CommitLog-3-1428238370079.log

java    5490 cassandra  DEL    REG              252,0             3670465 /usr/share/dse/common/log4j-1.2.17.jar

java    5490 cassandra  DEL    REG              252,0             3670467 /usr/share/dse/common/slf4j-log4j12-1.7.2.jar


java    5490 cassandra  439w   REG              252,0   120225    3802518 /var/log/cassandra/system.log


Cassandra DSE did occupy this port. What do I do wrong???

I have three nodes and agents work there witout any problems

Reply | Threaded
Open this post in threaded view
|

Re: DSE 4.6 with OpsCenter 5.1.1, agent can't start, port 9042 is occupied by DSE

DuyHai Doan
I think the problem is with the IP address. Cassandra does listen on  192.168.56.30 and you agent log complains about not being able to connect to 127.0.0.1.  Worth investigating there

On Sun, Apr 5, 2015 at 3:47 PM, Serega Sheypak <[hidden email]> wrote:
Hi, getting weird problem when agent to connect to OpsCenter
OpsCenter installed on VM with DSE and agent.
It's not for production, I have 3 VMs with DSE and OpsCenter for dev/test purposes.

The stacktrace from agent log is:

vagrant@dsenode03:~$ sudo cat /var/log/datastax-agent/agent.log

 Starting DataStax agent monitor datastax_agent_monitor

 INFO [main] 2015-04-05 13:32:31,594 Loading conf files: /var/lib/datastax-agent/conf/address.yaml

 INFO [main] 2015-04-05 13:32:31,642 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_76

 INFO [main] 2015-04-05 13:32:31,642 DataStax Agent version: 5.1.1

 INFO [main] 2015-04-05 13:32:31,679 Default config values: {:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings", :restore_req_update_period 60, :my_channel_prefix "/agent", :poll_period 60, :thrift_conn_timeout 10000, :rollups60_ttl 604800, :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :max-seconds-to-sleep 25, :private-conf-props ["initial_token" "listen_address" "broadcast_address" "rpc_address"], :thrift_port 9160, :async_retry_timeout 5, :agent-conf-group "global-cluster-agent-group", :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1, :async_queue_size 5000, :backup_staging_dir nil, :read-buffer-size 10000000, :remote_verify_max 300000, :disk_usage_update_period 60, :throttle-bytes-per-second 500000, :rollups7200_ttl 31536000, :remote_backup_retries 3, :ssl_keystore nil, :rollup_snapshot_period 300, :is_package true, :monitor_command "/usr/share/datastax-agent/bin/datastax_agent_monitor", :thrift_socket_timeout 5000, :remote_verify_initial_delay 1000, :cassandra_log_location "/var/log/cassandra/system.log", :remote_backup_region "us-west-1", :restore_on_transfer_failure false, :tmp_dir "/var/lib/datastax-agent/tmp/", :config_md5 nil, :jmx_port 7199, :write-buffer-size 100000, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl 0, :nodedetails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :backup_file_queue_max 10000, :jmx_thread_pool_size 5, :production 1, :runs_sudo 1, :max_file_transfer_attempts 30, :stomp_interface nil, :storage_keyspace "OpsCenter", :hosts ["127.0.0.1"], :rollup_snapshot_threshold 300, :jmx_retry_timeout 30, :unthrottled-default 10000000000, :remote_backup_retry_delay 5000, :remote_backup_timeout 1000, :seconds-to-read-kill-channel 0.005, :realtime_interval 5, :pdps_ttl 259200}

 INFO [main] 2015-04-05 13:32:31,924 Waiting for the config from OpsCenter

 INFO [main] 2015-04-05 13:32:31,925 Attempting to determine Cassandra's broadcast address through JMX

 INFO [Initialization] 2015-04-05 13:32:31,926 New JMX connection (127.0.0.1:7199)

 INFO [main] 2015-04-05 13:32:31,947 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}

 INFO [Jetty] 2015-04-05 13:32:32,026 Jetty server started

 INFO [Initialization] 2015-04-05 13:32:32,054 Using 192.168.56.30 as the cassandra broadcast address

 INFO [Initialization] 2015-04-05 13:32:32,135 cassandra RPC address is  nil

 INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC address is  192.168.56.30

 INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC broadcast address is  192.168.56.30

ERROR [Initialization] 2015-04-05 13:32:32,342 Can't connect to Cassandra, retrying

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot connect))

at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)

at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)

at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)

at com.datastax.driver.core.Cluster.init(Cluster.java:158)

at com.datastax.driver.core.Cluster.connect(Cluster.java:246)

at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)

at clojure.lang.RestFn.invoke(RestFn.java:410)

at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)

at opsagent.cassandra$setup_cassandra$fn__2491.invoke(cassandra.clj:269)

at again.core$with_retries_STAR_$fn__2363.invoke(core.clj:98)

at again.core$with_retries_STAR_.invoke(core.clj:97)

at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:267)

at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:152)

at opsagent.jmx$determine_ip.invoke(jmx.clj:276)

at opsagent.jmx$setup_jmx$fn__2867.invoke(jmx.clj:293)

at clojure.lang.AFn.run(AFn.java:24)

at java.lang.Thread.run(Thread.java:745)


And it tries to repeat in several times.


I did:

vagrant@dsenode03:~$ sudo netstat -alnpt | grep 9042

tcp        0      0 192.168.56.30:9042      0.0.0.0:*               LISTEN      5490/java       


and see that some stuff already listening to this port

# cut some output

vagrant@dsenode03:~$ sudo lsof -p 5490

COMMAND  PID      USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME

java    5490 cassandra  cwd    DIR              252,0     4096    3670233 /usr/share/dse

java    5490 cassandra  rtd    DIR              252,0     4096          2 /


# cut some output

vagrant@dsenode03:~$ sudo lsof -p 5490 | grep .log     

java    5490 cassandra  mem    REG              252,0 33554432    3803832 /var/lib/cassandra/commitlog/CommitLog-3-1428238370080.log

java    5490 cassandra  mem    REG              252,0 33554432    3803355 /var/lib/cassandra/commitlog/CommitLog-3-1428238370079.log

java    5490 cassandra  DEL    REG              252,0             3670465 /usr/share/dse/common/log4j-1.2.17.jar

java    5490 cassandra  DEL    REG              252,0             3670467 /usr/share/dse/common/slf4j-log4j12-1.7.2.jar


java    5490 cassandra  439w   REG              252,0   120225    3802518 /var/log/cassandra/system.log


Cassandra DSE did occupy this port. What do I do wrong???

I have three nodes and agents work there witout any problems