Question by Shawky Foda · Nov 27, 2018 at 08:34 PM · 126 Views Private Cloud organization monetize

Error during enable monetization for an organization : Error while accessing datastore;Please retry later

Following instruction in : 

https://docs.apigee.com/api-platform/monetization/enabling-monetization-organization

We are facing a problem while enabling monetization for an organization as below : 

Request for enable monetization To Management API: 
 /v1/mint/asyncjobs/enablemonetization

{
   "orgName" : "orgname" ,
   "mxGroup" : "mxgroup001",
   "pgHostName" : "apigeeaioserver.orgname.net",
   "pgPort" : "5432",
   "pgUserName" : "postgre",
   "pgPassword" : "Master@123",
   "adminEmail" : "sfoda@orgname.net",
   "notifyTo" : "sfoda@orgname.net",
   "taxEngineExternalId" : ""
}
---------------------------Response ----------------------
{
    "id": "532a98c9-f26a-4500-9053-01deec12e881",
    "log": "",
    "orgId": "orgname",
    "status": "RUNNING",
    "type": "ENABLE_MINT"
}

When checking the the status of the request by issuing a GET to /asyncjobs/{id}: 

{
    "id": "532a98c9-f26a-4500-9053-01deec12e881",
    "log": "Error while accessing datastore;Please retry later",
    "orgId": "orgname",
    "status": "FAILED",
    "type": "ENABLE_MINT"
}

By Investigating /opt/apigee/var/log/edge-management-server/logs/system.log, I found the below errors : 

2018-11-27 20:23:08,282  Apigee-Main-23 INFO  MINT.SERVICE - OrgSetupFacadeImpl.setup() : Completed :com.apigee.mint.setup.SeedBillingDocumentNumbersStep
2018-11-27 20:23:08,282  Apigee-Main-23 INFO  c.a.m.a.AsyncProcessor - AsyncProcessor$1.pre() : listener pre [] 532a98c9-f26a-4500-9053-01deec12e881
2018-11-27 20:23:08,283  Apigee-Main-23 INFO  MINT.SERVICE - OrgSetupFacadeImpl.setup() : Executing :com.apigee.mint.setup.EnableNotificationStep
2018-11-27 20:23:08,348  Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.searchForRecordsWithCriteriaList() :  Couldn't search for specific criteria  
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:26) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:745) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:727) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:132) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getIndexedSlices(KeyspaceServiceImpl.java:749) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:148) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:139) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:90) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery.execute(IndexedSlicesQuery.java:138) ~[hector-core-2.0-0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:417) [datastore-service-1.0.0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) [datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) [mint-1.170821.01-SNAPSHOT.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20464) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20422) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result.read(Cassandra.java:20348) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
	at org.apache.cassandra.thrift.Cassandra$Client.recv_get_indexed_slices(Cassandra.java:881) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$Client.get_indexed_slices(Cassandra.java:865) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:733) ~[hector-core-2.0-0.jar:na]
	... 25 common frames omitted
2018-11-27 20:23:08,350  Apigee-Main-23 ERROR c.a.m.a.AsyncOrgSetup - AsyncOrgSetup.onError() : Error processing offlinejob com.apigee.mint.model.Job@55b618a8[orgId=orgname,status=RUNNING,type=ENABLE_MINT,log=,id=532a98c9-f26a-4500-9053-01deec12e881,created=Tue Nov 27 20:23:07 UTC 2018,createdBy=<null>,updated=Tue Nov 27 20:23:07 UTC 2018,updatedBy=<null>]
com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:464) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
2018-11-27 20:23:08,352  Apigee-Main-23 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist
2018-11-27 20:23:08,390  Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.insert() :  @@@@@@@@@@@@@@ CREATING OPERATION FAILED !!!!!!
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:102) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69) ~[hector-core-2.0-0.jar:na]
	at com.apigee.datastore.client.CassandraClient.insert(CassandraClient.java:144) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.insertNotificationServiceItem(NotificationCassandraDataStore.java:116) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.addNotificationEmailMessage(NotificationFacade.java:147) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.sendNotificationEmailMessage(NotificationFacade.java:131) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.util.EmailUtil.scheduleEmail(EmailUtil.java:95) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.NotifyOnCompletionStep.execute(NotifyOnCompletionStep.java:36) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncProcessor.notifyJobStatus(AsyncProcessor.java:88) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncProcessor.updateAndNotify(AsyncProcessor.java:75) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.onError(AsyncOrgSetup.java:62) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.threadpool.CallbackTask.done(CallbackTask.java:44) [threadpool-1.0.0.jar:na]
	at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.setException(FutureTask.java:251) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:271) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28411) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28388) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:28322) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
	at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1078) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1064) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:268) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
	... 23 common frames omitted
2018-11-27 20:23:34,426  qtp1200546947-91 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist


Could anyone advise what is the root cause and how to overcome? 

@anilsagar

Add comment Show 1

10 |5000 characters needed characters left characters exceeded

Shawky Foda · Feb 04, 2019 at 05:27 AM 0

Link

Steps to Resolve issues due to installing monitization in AIO :

$ cd /opt/apigee/apigee-cassandra-2.1.16-0.0.1123/bin
$ ./cqlsh 192.168.56.103 9042

cqlsh>SELECT * FROM system.schema_keyspaces ;

You will notice that the replication factor = 3 for mint , taurus , notification keyspaces ....

keyspace_name  | durable_writes | strategy_class  | strategy_options
------------------------+----------------+------------------------------------------------------+----------------------------
  auth |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"1"}
  notification |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"3"}
  cache |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"1"}
taurus |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"3"}
devconnect |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"1"}
system_traces
|  True |  org.apache.cassandra.locator.SimpleStrategy |
{"replication_factor":"2"}
 
…….
keyvaluemap |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"1"}
 mint |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":“3"}
  counter |  True | org.apache.cassandra.locator.NetworkTopologyStrategy
| 
{"dc-1":"1"}
cqlsh>
alter keyspace mint
WITH REPLICATION = {'class': 'NetworkTopologyStrategy' ,
'dc-1':'1' }; 
cqlsh>
alter keyspacetaurus WITH
REPLICATION = {'class': 'NetworkTopologyStrategy' ,
'dc-1':'1' }; 
cqlsh>
alter keyspace
notification WITH REPLICATION = {'class': 'NetworkTopologyStrategy' ,
'dc-1':'1' };