VHOST ports are not listening on router

Hi Apigeeks,

I have created a VHOST in my "org" and environment "env" for normal HTTP and HTTPS traffic. But when I do an api call to the router with given port, I am getting a connection refused status. I even restarted router after creating VHOST.

I checked the netstat -ltnp and couldn't find any VHOST ports listening.

Checked for /opt/nginx/conf.d/ and no conf files are present for the VHOST there.

In router logs I found an something like below:

2017-06-08 10:43:14,187  pool-9-thread-5 INFO  BOOTSTRAP - RuntimeConfigurationServiceImpl.internalConfigureEnvironment() : No bootstrap listeners found for deploy events, skipping application deployments
2017-06-08 10:43:14,187  pool-9-thread-6 ERROR BOOTSTRAP - RuntimeConfigurationServiceImpl.dispatchToListeners() : RuntimeConfigurationServiceImpl.dispatchToListeners : Error occurred while dispatching the requests [ConfigEvent{beanName='null', path='/organizations/org/environments/env/', action=sync, beanClass=class com.apigee.messaging.config.beans.EnvironmentBean, bean=null, itemType='null', description='SYNC Environment /organizations/org/environments/env/', xid=null}] to com.apigee.application.bootstrap.listeners.RouterManagementBootstrapListener@6831d540
com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later
        at com.apigee.datastore.client.astyanax.AstyanaxCassandraClient.fetchColumnValueAsByteArray(AstyanaxCassandraClient.java:1469) ~[datastore-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.readFromCassandra(CassandraRepositoryDelegate.java:662) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.readFromEnvironmentsColumnFamily(CassandraRepositoryDelegate.java:597) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.loadFile(CassandraRepositoryDelegate.java:498) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepository.loadFile(CassandraRepository.java:97) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.RepositoryServiceImpl.load(RepositoryServiceImpl.java:298) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.beans.BeanUtil.load(BeanUtil.java:91) ~[beans-1.0.0.jar:na]
        at com.apigee.messaging.config.ConfigEvent.getBean(ConfigEvent.java:127) ~[config-model-1.0.0.jar:na]

We are on OPDK version 4.16.09.

Solved Solved
1 5 590
1 ACCEPTED SOLUTION

Hi All,

Resolved this issue.

Reason:

Our infra team accidentally made a clone of one of the apigee cluster node which got registered itself in zookeeper conf. This also resulted in starting of components in the wrong sequence(this cloned nodes had analytics and central components).

Also this had impact on the analytics not being pulled in UI dashboards.

Fix:

Stopped all the components on new cloned nodes.

Stopped all other components of apigee installation in proper stopping sequence as documented in Apigee Operations Guide.

Restarted all the components in proper starting sequence.

For reference Start/Stop Order.

View solution in original post

5 REPLIES 5

Not applicable

Hi @Mohammed Zuber,

I think the APIproxies were not deployed as per this statement

No bootstrap listeners found for deploy events, skipping application deployments

Before we investigate why that happened, I see the below statement which tells us that there are some issues with the wiring

com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later 

Looks like you are having connectivity issues and especially connectivity to Cassandra.
Let's fix that first.

You can check the wiring status by running the below APIs or you can share here with us if you are okay with that

Curl -v http://mgmthost:8080/v1/servers?pod=central&region=dc01
Curl -v http://mgmthost:8080/v1/servers?pod=gateway&region=dc01 
Curl -v http://mgmthost:8080/v1/servers?pod=analytics&region=dc01 

Also make sure you have all the required ports open as per this http://docs.apigee.com/private-cloud/latest/installation-requirements

Thanks @Maruti Chand,

I have checked for wiring, the connections seems to be okay. All ports are open and statuses are green in servers.

We have noticed that in the system.log of a working and non-working roters there is this difference: also, we checked the ports and they are all open from all R to all CS.: zk 2181, and cs 9042 9160

Is there any way to establish why com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException happens?

working Router:

2017-06-08 12:45:55,710  main INFO  DATASTORE.CASSANDRA - ConnectionPoolManager.createConnectionPoolKey() : created co
nnection pool key apprepo#<CS_IP>_<CS_IP>_<CS_IP> for config name application-datastore
2017-06-08 12:45:55,866  main INFO  c.n.a.c.i.ConnectionPoolMBeanManager - ConnectionPoolMBeanManager.registerMonitor(
) : Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=application-datastore,ServiceType=connectionp
ool
2017-06-08 12:45:55,875  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
2017-06-08 12:45:55,876  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
2017-06-08 12:45:55,876  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
--->
2017-06-08 12:45:56,233  main INFO  BOOTSTRAP - RuntimeConfigurationServiceImpl$1.run() : RuntimeConfigurationServiceI
mpl.start : Server is registered as [router] Will proceed to sync up the configurations
2017-06-08 12:45:56,233  main INFO  BOOTSTRAP - RuntimeConfigurationServiceImpl.initiateSync() : RuntimeConfigurationS
erviceImpl.initiateSync : async : true
2017-06-08 12:45:56,236  main INFO  BOOTSTRAP - RuntimeConfigurationServiceImpl.internalInitiateSync() : RuntimeConfig
urationServiceImpl.internalInitiateSync : Organizations : [org, VALIDATE]



non-working Router:


2017-06-08 12:00:36,221  main INFO  DATASTORE.CASSANDRA - ConnectionPoolManager.createConnectionPoolKey() : created co
nnection pool key apprepo#<CS_IP>_<CS_IP>_<CS_IP> for config name application-datastore
2017-06-08 12:00:36,420  main INFO  c.n.a.c.i.ConnectionPoolMBeanManager - ConnectionPoolMBeanManager.registerMonitor(
) : Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=application-datastore,ServiceType=connectionp
ool
2017-06-08 12:00:36,434  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
2017-06-08 12:00:36,435  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
2017-06-08 12:00:36,435  main INFO  c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde
d() : AddHost: <CS_IP>
---->
2017-06-08 12:00:48,511  main ERROR S.REPOSITORY.CASSANDRA - ApplicationRepoServerListener.initializeKeySpace() : Conn
ectionException while creating column family in apprepo keyspace for cluster:[application-datastore]
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=None(<CS_IP>):0, late
ncy=6001(6001), attempts=3]Timed out waiting for connection
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPoo
l.java:231) ~[astyanax-core-1.56.43.jar:na]
        at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool
.java:198) ~[astyanax-core-1.56.43.jar:na]
        at com.netflix.astyanax.connectionpool.impl.LeastOutstandingExecuteWithFailover.borrowConnection(LeastOutstand
ingExecuteWithFailover.java:74) ~[astyanax-core-1.56.43.jar:na]
        at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFa
iloverImpl.java:117) ~[astyanax-core-1.56.43.jar:na]
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHo
stPartitionConnectionPool.java:338) ~[astyanax-core-1.56.43.jar:na]
        at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:493) ~[astyanax-thr
ift-1.56.43.jar:na]
        at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.describeKeyspace(ThriftKeyspaceImpl.java:244) ~[astyanax-thr
ift-1.56.43.jar:na]
        at com.apigee.services.repository.nosql.ApplicationRepoServerListener.initializeKeySpace(ApplicationRepoServer
Listener.java:153) ~[repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.start(CassandraRepositoryDelegate.java:180
) [repository-impl-1.0.0.jar:na]
        at com.apigee.services.repository.nosql.CassandraRepository.start(CassandraRepository.java:33) [repository-imp
l-1.0.0.jar:na]
<br>

You mentioned all the servers are in green but can you check if your cassandra nodes are listening on 9160?

Do you have any properties defined in /opt/apigee/customer/application/cassandra.properties?

Yes all cassandra nodes are listening on port 9160 and we don't have any custom properties defined in /opt/apigee/customer/application/cassandra.properties

Hi All,

Resolved this issue.

Reason:

Our infra team accidentally made a clone of one of the apigee cluster node which got registered itself in zookeeper conf. This also resulted in starting of components in the wrong sequence(this cloned nodes had analytics and central components).

Also this had impact on the analytics not being pulled in UI dashboards.

Fix:

Stopped all the components on new cloned nodes.

Stopped all other components of apigee installation in proper stopping sequence as documented in Apigee Operations Guide.

Restarted all the components in proper starting sequence.

For reference Start/Stop Order.