Apigee Edge Management Server failing after startup

sam-was
Participant II

We are able to start the Mgmt Server but it is going down in a while with below exceptions.

Version: 4.17.0.5 - On Premise Installation

2018-01-05 12:56:06,948  main WARN  DATASTORE.CASSANDRA - 
ConfigurationUtil.resolveLocalDataCenter() : TException while resolving 
datacenter for 
datastoreconfig:[DataStoreConfig{name='identityzone-datastore', 
type=null, keyspace='identityzone', consistencyLevelPolicy=null, 
maxActiveConnections=10, thriftSocketTimeoutInMillis=0, 
dataStoreNodeConfigs=[DataStoreNodeConfig{host='XXXXXXXX', 
userName='null', password='null', uuid=XXXXXX---DDDDFFS-DDSS-0234, 
properties=null}, DataStoreNodeConfig{host='XX.XXX.XX.XXX', 
userName='null', password='null', 
uuid=6XXXXXXX-d4XXX-4ae4-9eXXXESDSD54d590, properties=null}, 
DataStoreNodeConfig{host='XX.XXX.XX.XXX', userName='null', 
password='null', uuid=dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, 
properties=null}]}]
org.apache.cassandra.thrift.InvalidRequestException: null

  at 
org.apache.cassandra.thrift.Cassandra$describe_ring_result$describe_ring_resultStandardScheme.read(Cassandra.java:35320)
 ~[cassandra-thrift-2.1.2.jar:2.1.2]
  at 
org.apache.cassandra.thrift.Cassandra$describe_ring_result$describe_ring_resultStandardScheme.read(Cassandra.java:35287)
 ~[cassandra-thrift-2.1.2.jar:2.1.2]
  at org.apache.cassandra.thrift.Cassandra$describe_ring_result.read(Cassandra.java:35229) ~[cassandra-thrift-2.1.2.jar:2.1.2]
  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
  at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_ring(Cassandra.java:1292) ~[cassandra-thrift-2.1.2.jar:2.1.2]
  at org.apache.cassandra.thrift.Cassandra$Client.describe_ring(Cassandra.java:1279) ~[cassandra-thrift-2.1.2.jar:2.1.2]

  at 
com.apigee.datastore.util.ConfigurationUtil.resolveLocalDataCenter(ConfigurationUtil.java:69)
 ~[datastore-service-1.0.0.jar:na]
  at 
com.apigee.datastore.pooling.CqlDriverConnectionPool.createSession(CqlDriverConnectionPool.java:115)
 [datastore-service-1.0.0.jar:na]
  at 
com.apigee.datastore.pooling.CqlDriverConnectionPool.<init>(CqlDriverConnectionPool.java:36)
 [datastore-service-1.0.0.jar:na]
  at com.apigee.datastore.DataStoreServiceImpl.createInstance(DataStoreServiceImpl.java:182) [datastore-service-1.0.0.jar:na]
  at com.apigee.zones.datastore.IdentityZonesDataStore.init(IdentityZonesDataStore.java:166) [identity-zones-1.0.0.jar:na]
  at com.apigee.registration.events.BasicDispatcher.registerListener(BasicDispatcher.java:180) [registration-1.0.0.jar:na]

  at 
com.apigee.registration.ServerRegistrationServiceImpl.register(ServerRegistrationServiceImpl.java:608)
 [registration-1.0.0.jar:na]
  at 
com.apigee.registration.ServerRegistrationServiceImpl.register(ServerRegistrationServiceImpl.java:593)
 [registration-1.0.0.jar:na]
  at com.apigee.zones.datastore.IdentityZonesDataStore.init(IdentityZonesDataStore.java:90) [identity-zones-1.0.0.jar:na]
  at com.apigee.zones.datastore.DataStoreFactory.init(DataStoreFactory.java:53) [identity-zones-1.0.0.jar:na]
  at com.apigee.zones.service.IdentityZoneServiceImpl.start(IdentityZoneServiceImpl.java:51) [identity-zones-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:168) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:71) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:185) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.start(MicroKernel.java:146) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.start(MicroKernel.java:141) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.main(MicroKernel.java:91) [microkernel-1.0.0.jar:na]

     
     
2018-01-05
 12:56:35,622 org:xxxxxxx  pool-5-thread-8 ERROR K.DATASTORE.NOSQL - 
CacheUtil.populateLowerCaseToProperCaseMappingForOrg() : Error adding to
 cache - ignoring the exception -
java.lang.NullPointerException: null

  at 
com.apigee.keymanagement.util.CacheUtil.populateLowerCaseToProperCaseMappingForOrg(CacheUtil.java:228)
 ~[keymanagement-1.0.0.jar:na]
  at 
com.apigee.keymanagement.MetaDataInvocationListener.beforeInvocation(MetaDataInvocationListener.java:46)
 [keymanagement-1.0.0.jar:na]
  at com.apigee.rest.framework.CustomJAXRSInvoker.beforeInvocation(CustomJAXRSInvoker.java:163) [rest-1.0.0.jar:na]
  at com.apigee.rest.framework.CustomJAXRSInvoker.performInvocation(CustomJAXRSInvoker.java:118) [rest-1.0.0.jar:na]
  at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) [cxf-rt-core-2.4.7.jar:2.4.7]
  at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
  at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:241) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
  at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
  at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-rt-core-2.4.7.jar:2.4.7]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112]
  at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-rt-core-2.4.7.jar:2.4.7]

  at 
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
 [cxf-rt-core-2.4.7.jar:2.4.7]
  at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.4.7.jar:2.4.7]
  at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-rt-core-2.4.7.jar:2.4.7]

  at 
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:232)
 [cxf-rt-transports-http-2.4.10.jar:2.4.7]
  at 
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:209)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at 
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:152)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at 
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:114)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:113)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]

  at 
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
 [cxf-rt-transports-http-2.4.10.jar:2.4.10]
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:830) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]

  at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543)
 [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
  at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:530)
 [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
  at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
 [jetty-security-9.4.0.v20161208.jar:9.4.0.v20161208]
   
2018-01-05
 12:56:37,610  main ERROR KERNEL.DEPLOYMENT - 
ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life 
cycle exception while starting service [IdentityZoneService, All host(s)
 tried for query failed (tried: /XX.XXX.XXX.5:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.5:9042] 
Cannot connect), /XX.XXX.XXX.7:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.7:9042] 
Cannot connect), /XX.XXX.XXX.6:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.6:9042] 
Cannot connect))] : {}
com.datastax.driver.core.exceptions.NoHostAvailableException:
 All host(s) tried for query failed (tried: /XX.XXX.XXX.5:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.5:9042] 
Cannot connect), /XX.XXX.XXX.7:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.7:9042] 
Cannot connect), /XX.XXX.XXX.6:9042 
(com.datastax.driver.core.TransportException: [/XX.XXX.XXX.6:9042] 
Cannot connect))
  at 
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
 ~[apigee-cassandra-driver-core-2.1.4.jar:na]
  at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) ~[apigee-cassandra-driver-core-2.1.4.jar:na]
  at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230) ~[apigee-cassandra-driver-core-2.1.4.jar:na]
  at com.datastax.driver.core.Cluster.init(Cluster.java:157) ~[apigee-cassandra-driver-core-2.1.4.jar:na]
  at com.datastax.driver.core.Cluster.connect(Cluster.java:245) ~[apigee-cassandra-driver-core-2.1.4.jar:na]
  at com.datastax.driver.core.Cluster.connect(Cluster.java:278) ~[apigee-cassandra-driver-core-2.1.4.jar:na]

  at 
com.apigee.datastore.pooling.CqlDriverConnectionPool.createSession(CqlDriverConnectionPool.java:135)
 ~[datastore-service-1.0.0.jar:na]
  at 
com.apigee.datastore.pooling.CqlDriverConnectionPool.<init>(CqlDriverConnectionPool.java:36)
 ~[datastore-service-1.0.0.jar:na]
  at com.apigee.datastore.DataStoreServiceImpl.createInstance(DataStoreServiceImpl.java:182) ~[datastore-service-1.0.0.jar:na]
  at com.apigee.zones.datastore.IdentityZonesDataStore.init(IdentityZonesDataStore.java:166) ~[identity-zones-1.0.0.jar:na]
  at com.apigee.registration.events.BasicDispatcher.registerListener(BasicDispatcher.java:180) ~[registration-1.0.0.jar:na]

  at 
com.apigee.registration.ServerRegistrationServiceImpl.register(ServerRegistrationServiceImpl.java:608)
 ~[registration-1.0.0.jar:na]
  at 
com.apigee.registration.ServerRegistrationServiceImpl.register(ServerRegistrationServiceImpl.java:593)
 ~[registration-1.0.0.jar:na]
  at com.apigee.zones.datastore.IdentityZonesDataStore.init(IdentityZonesDataStore.java:90) ~[identity-zones-1.0.0.jar:na]
  at com.apigee.zones.datastore.DataStoreFactory.init(DataStoreFactory.java:53) ~[identity-zones-1.0.0.jar:na]
  at com.apigee.zones.service.IdentityZoneServiceImpl.start(IdentityZoneServiceImpl.java:51) ~[identity-zones-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:168) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:71) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]

  at 
com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:356)
 [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:185) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.start(MicroKernel.java:146) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.start(MicroKernel.java:141) [microkernel-1.0.0.jar:na]
  at com.apigee.kernel.MicroKernel.main(MicroKernel.java:91) [microkernel-1.0.0.jar:na]

Solved Solved
0 12 1,878
2 ACCEPTED SOLUTIONS

If the issue was connectivity which prevented the keyspace from being created at upgrade time, you can rerun the management server setup script and it should correct any gaps in the database schema:

/opt/apigee/apigee-setup/bin/setup.sh -f /tmp/apigee-response.txt -p ms

View solution in original post

sam-was
Participant II

Initial error

Initial error was due to the fact that we never had openings (for port 9042) from our MGMT server to the Zookeeper/Cassandra node.

This is due to latest upgrade of Cassandra, all clients connect on 9042 now. In the past, the Thrift port of 9160 was used.

After opening of FW

Once we opened the firewall the mgmt server was able to make connection, but it was complaining about the missing keyspace.

This keyspace "identityzone" is a newly created keyspace only used when if you are using the SSO for authentication. Though it was not used in our case but still it made our setup to go down.

So we created the keyspace and the tables in that.

CREATE KEYSPACE identityzone WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1': '1'}  AND durable_writes = true;

CREATE TABLE identityzone.identityzones (
    name text PRIMARY KEY,
    organizations set<text>
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

CREATE TABLE identityzone.orgtoidentityzone (
    organization text PRIMARY KEY,
    zonename text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE'

And it worked. 🙂

Thanks @potturiprasanth and @paulmibus

for all the support.

View solution in original post

12 REPLIES 12

Could you please verify below

- Whether you are able to connect to all Cassandra IPs on 9042 port from Management Server

- Verify all Cassandra servers status

- IP address provided in the installation configFile '/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile'

Could you please verify below

- Whether you are able to connect to all Cassandra IPs on 9042 port from Management Server

Sam: We have FW for 9042 port but it was never open from the mgmt server. RPC port 9160 is working ok and FW are open.

- Verify all Cassandra servers status

Sam: All looks ok and Cassandra is reply to the live traffic only mgmt server is not coming up.

- IP address provided in the installation configFile '/opt/apigee/apigee-setup/bin/setup.sh -p component -f configFile'

Sam: IP addresses are correct.

🙂 Thanks for the help. Just for info.

Not a fresh install but upgraded few weeks back and this is first restart of mgmt server after the upgrade. No problem with any live traffic only the mgmt server is not coming up.

Regards,

Saman

Is this a fresh installation? If not, was the management server functioning normally before now? Have any changes been made to the configuration recently?

If this is a fresh installation, can you post the values you entered for CASS_HOSTS in the response file?

Hi,

Not a fresh install but upgraded few weeks back and this is first restart of mgmt server after the upgrade. No problem with any live traffic only the mgmt server is not coming up.

No changes have been made to the configuration recently apart from regular proxy deploys.

Thanks for the help. 🙂

Regards

Saman

sam-was
Participant II

Is it possible that the UUID is not matching with latest UUID for cassandra?

How can I check the UUID for cassandra?

host='XX.XXX.XX.XXX', userName='null', 
password='null', uuid=dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Apart fro that I see below error

ConfigurationUtil.resolveLocalDataCenter():TExceptionwhile resolving 
datacenter for 
datastoreconfig:[DataStoreConfig{name='identityzone-datastore', 
type=null, keyspace='identityzone'

I don't see this keyspace at all in the Cassandra, is this some new datastore?

Could you please try running below

--Run on one of cassandra node

Run the nodetool ring command on the modified node to ensure that the ring is complete. The utility can be found at <inst_root>/apigee/apigee­cassandra/bin .

> nodetool ­h localhost ring

- On each of cassandra node

Run nodetool repair on the modified node. Note that this process may take some time, so it is highly recommended that this not be done during peak API traffic hours.

> nodetool ­h <IP Address of Cassandra server> repair

- Try restarting edge management server

Thanks,

@potturiprasanth

Thanks. I will come up with the details on your suggestions.

Hi,


We opened the firewall from MGMT server to 9042 and it is able to make connection but now problem is that the keyspace "identityzone" (introduced in latest version) doesn't exist at all in Cassandra. So I think we need to create the keyspace, but what about the tables in that, will MGMT be able to make them by itself?

018-01-09 10:43:55,140  main INFO  c.d.driver.core.Cluster - Cluster$Manager.init() : New Cassandra host /XX.XXX.XX.5:9042 added
2018-01-09 10:43:55,140  main INFO  c.d.driver.core.Cluster - Cluster$Manager.init() : New Cassandra host /XX.XXX.XX.7:9042 added
2018-01-09 10:43:55,141  main INFO  c.d.driver.core.Cluster - Cluster$Manager.init() : New Cassandra host /XX.XXX.XX.6:9042 added
2018-01-09 10:43:55,769  main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [IdentityZoneService, Keyspace 'identityzone' does not exist] : {}
com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'identityzone' does not exist 

Regards,

Saman

Could you please confirm whether the management server is running now..

if yes then please verify whether below types are configured for each of the pods

POD-Central

● curl ­u $ADMINEMAIL:$PW "http://$MSIP:$port/v1/servers?pod=central" | egrep ­i '(type|internalip|uuid|region)'

type=scheduler-datastore

type=audit-datastore

type=apimodel-datastore

type=application-datastore

type=edgenotification-datastore

type=identityzone-datastore

type=user-settings-datastore

type=auth-datastore

POD- Analytics

● curl ­u $ADMINEMAIL:$PW "http://$MSIP :$port/v1/servers?pod=analytics" | egrep ­i '(type|internalip|uuid|region)'

type=analytics-datastore

type=reportcrud-datastore

POD-Gateway

● curl ­u $ADMINEMAIL:$PW "http://$MSIP: $port/v1/servers?pod=gateway" | egrep ­i '(type|internalip|uuid|region)'

type=keyvaluemap-datastore

type=cache-datastore

type=counter-datastore

type=kms-datastore

type=dc-datastore

If you notice some of the type are missing, you can add the keystores by management API calls, as below

POD-Gateway

curl -v ­u $ADMINEMAIL:$PW http://$MSIP:$port/v1/servers -X POST -d "type=keyvaluemap-datastore&type=cache-datastore&type=counter-datastore&type=kms-datastore&type=dc-datastore&region={regionName}&pod={gatewayPodName}&InternalIP={cassandraIP/s}"

POD-Analytics

curl -v ­u $ADMINEMAIL:$PW http://$MSIP:$port/v1/servers -X POST -d "type=analytics-datastore&type=reportcrud-datastore&region={regionName}&pod=analytics&InternalIP={cassandraIP/s}"

POD-Central

curl -v -u $ADMINEMAIL:$PW http://$MSIP:$port/v1/servers -X POST -d "type=scheduler-datastore&type=audit-datastore&type=apimodel-datastore&type=application-datastore&type=edgenotification-datastore&type=identityzone-datastore&type=user-settings-datastore&type=auth-datastore&region={regionName}&pod=central&InternalIP={cassandraIP/s}"

Thanks,

If the issue was connectivity which prevented the keyspace from being created at upgrade time, you can rerun the management server setup script and it should correct any gaps in the database schema:

/opt/apigee/apigee-setup/bin/setup.sh -f /tmp/apigee-response.txt -p ms

I will sort of accept your answer. 🙂 But would like to give a way how we did it and other details.

Thanks a ton for the support.

sam-was
Participant II

Initial error

Initial error was due to the fact that we never had openings (for port 9042) from our MGMT server to the Zookeeper/Cassandra node.

This is due to latest upgrade of Cassandra, all clients connect on 9042 now. In the past, the Thrift port of 9160 was used.

After opening of FW

Once we opened the firewall the mgmt server was able to make connection, but it was complaining about the missing keyspace.

This keyspace "identityzone" is a newly created keyspace only used when if you are using the SSO for authentication. Though it was not used in our case but still it made our setup to go down.

So we created the keyspace and the tables in that.

CREATE KEYSPACE identityzone WITH replication = {'class': 'NetworkTopologyStrategy', 'dc-1': '1'}  AND durable_writes = true;

CREATE TABLE identityzone.identityzones (
    name text PRIMARY KEY,
    organizations set<text>
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

CREATE TABLE identityzone.orgtoidentityzone (
    organization text PRIMARY KEY,
    zonename text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE'

And it worked. 🙂

Thanks @potturiprasanth and @paulmibus

for all the support.