Error during enable monetization for an organization : Error while accessing datastore;Please retry later
Following instruction in :
https://docs.apigee.com/api-platform/monetization/enabling-monetization-organization
We are facing a problem while enabling monetization for an organization as below :
Request for enable monetization To Management API:
/v1/mint/asyncjobs/enablemonetization
{
"orgName" : "orgname" ,
"mxGroup" : "mxgroup001",
"pgHostName" : "apigeeaioserver.orgname.net",
"pgPort" : "5432",
"pgUserName" : "postgre",
"pgPassword" : "Master@123",
"adminEmail" : "sfoda@orgname.net",
"notifyTo" : "sfoda@orgname.net",
"taxEngineExternalId" : ""
}
---------------------------Response ----------------------
{
"id": "532a98c9-f26a-4500-9053-01deec12e881",
"log": "",
"orgId": "orgname",
"status": "RUNNING",
"type": "ENABLE_MINT"
}
When checking the the status of the request by issuing a GET to /asyncjobs/{id}:
{
"id": "532a98c9-f26a-4500-9053-01deec12e881",
"log": "Error while accessing datastore;Please retry later",
"orgId": "orgname",
"status": "FAILED",
"type": "ENABLE_MINT"
}
By Investigating /opt/apigee/var/log/edge-management-server/logs/system.log, I found the below errors :
2018-11-27 20:23:08,282 Apigee-Main-23 INFO MINT.SERVICE - OrgSetupFacadeImpl.setup() : Completed :com.apigee.mint.setup.SeedBillingDocumentNumbersStep
2018-11-27 20:23:08,282 Apigee-Main-23 INFO c.a.m.a.AsyncProcessor - AsyncProcessor$1.pre() : listener pre [] 532a98c9-f26a-4500-9053-01deec12e881
2018-11-27 20:23:08,283 Apigee-Main-23 INFO MINT.SERVICE - OrgSetupFacadeImpl.setup() : Executing :com.apigee.mint.setup.EnableNotificationStep
2018-11-27 20:23:08,348 Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.searchForRecordsWithCriteriaList() : Couldn't search for specific criteria
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:26) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:745) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:727) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:132) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getIndexedSlices(KeyspaceServiceImpl.java:749) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:148) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:139) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:90) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.IndexedSlicesQuery.execute(IndexedSlicesQuery.java:138) ~[hector-core-2.0-0.jar:na]
at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:417) [datastore-service-1.0.0.jar:na]
at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) [datastore-service-1.0.0.jar:na]
at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) [mint-1.170821.01-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20464) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20422) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result.read(Cassandra.java:20348) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.cassandra.thrift.Cassandra$Client.recv_get_indexed_slices(Cassandra.java:881) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$Client.get_indexed_slices(Cassandra.java:865) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:733) ~[hector-core-2.0-0.jar:na]
... 25 common frames omitted
2018-11-27 20:23:08,350 Apigee-Main-23 ERROR c.a.m.a.AsyncOrgSetup - AsyncOrgSetup.onError() : Error processing offlinejob com.apigee.mint.model.Job@55b618a8[orgId=orgname,status=RUNNING,type=ENABLE_MINT,log=,id=532a98c9-f26a-4500-9053-01deec12e881,created=Tue Nov 27 20:23:07 UTC 2018,createdBy=<null>,updated=Tue Nov 27 20:23:07 UTC 2018,updatedBy=<null>]
com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later
at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:464) ~[datastore-service-1.0.0.jar:na]
at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) ~[datastore-service-1.0.0.jar:na]
at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) ~[notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) ~[notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) ~[notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) ~[mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) ~[mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) ~[mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) ~[mint-1.170821.01-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
2018-11-27 20:23:08,352 Apigee-Main-23 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist
2018-11-27 20:23:08,390 Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.insert() : @@@@@@@@@@@@@@ CREATING OPERATION FAILED !!!!!!
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:102) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69) ~[hector-core-2.0-0.jar:na]
at com.apigee.datastore.client.CassandraClient.insert(CassandraClient.java:144) ~[datastore-service-1.0.0.jar:na]
at com.apigee.notification.nosql.NotificationCassandraDataStore.insertNotificationServiceItem(NotificationCassandraDataStore.java:116) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.service.NotificationFacade.addNotificationEmailMessage(NotificationFacade.java:147) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.notification.service.NotificationFacade.sendNotificationEmailMessage(NotificationFacade.java:131) [notification-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.util.EmailUtil.scheduleEmail(EmailUtil.java:95) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.setup.NotifyOnCompletionStep.execute(NotifyOnCompletionStep.java:36) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.asyncjob.AsyncProcessor.notifyJobStatus(AsyncProcessor.java:88) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.asyncjob.AsyncProcessor.updateAndNotify(AsyncProcessor.java:75) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.mint.asyncjob.AsyncOrgSetup.onError(AsyncOrgSetup.java:62) [mint-1.170821.01-SNAPSHOT.jar:na]
at com.apigee.threadpool.CallbackTask.done(CallbackTask.java:44) [threadpool-1.0.0.jar:na]
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) [na:1.8.0_131]
at java.util.concurrent.FutureTask.setException(FutureTask.java:251) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:271) [na:1.8.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28411) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28388) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:28322) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1078) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1064) ~[cassandra-thrift-2.1.2.jar:2.1.2]
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:268) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
... 23 common frames omitted
2018-11-27 20:23:34,426 qtp1200546947-91 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist
Could anyone advise what is the root cause and how to overcome?
@anilsagar
