{ Community }
  • Academy
  • Docs
  • Developers
  • Resources
    • Community Articles
    • Apigee on GitHub
    • Code Samples
    • Videos & eBooks
    • Accelerator Methodology
  • Support
  • Ask a Question
  • Spaces
    • Product Announcements
    • General
    • Edge/API Management
    • Developer Portal (Drupal-based)
    • Developer Portal (Integrated)
    • API Design
    • APIM on Istio
    • Extensions
    • Business of APIs
    • Academy/Certification
    • Adapter for Envoy
    • Analytics
    • Events
    • Hybrid
    • Integration (AWS, PCF, Etc.)
    • Microgateway
    • Monetization
    • Private Cloud Deployment
    • 日本語コミュニティ
    • Insights
    • IoT Apigee Link
    • BaaS/Usergrid
    • BaaS Transition/Migration
    • Apigee-127
    • New Customers
    • Topics
    • Questions
    • Articles
    • Ideas
    • Leaderboard
    • Badges
  • Log in
  • Sign up

Get answers, ideas, and support from the Apigee Community

  • Home /
  • Monetization /
avatar image
1
Question by Shawky Foda · Nov 27, 2018 at 08:34 PM · 97 Views Private Cloudorganizationmonetize

Error during enable monetization for an organization : Error while accessing datastore;Please retry later

Following instruction in : 

https://docs.apigee.com/api-platform/monetization/enabling-monetization-organization

We are facing a problem while enabling monetization for an organization as below : 

Request for enable monetization To Management API: 
 /v1/mint/asyncjobs/enablemonetization

{
   "orgName" : "orgname" ,
   "mxGroup" : "mxgroup001",
   "pgHostName" : "apigeeaioserver.orgname.net",
   "pgPort" : "5432",
   "pgUserName" : "postgre",
   "pgPassword" : "Master@123",
   "adminEmail" : "sfoda@orgname.net",
   "notifyTo" : "sfoda@orgname.net",
   "taxEngineExternalId" : ""
}
---------------------------Response ----------------------
{
    "id": "532a98c9-f26a-4500-9053-01deec12e881",
    "log": "",
    "orgId": "orgname",
    "status": "RUNNING",
    "type": "ENABLE_MINT"
}

When checking the the status of the request by issuing a GET to /asyncjobs/{id}: 

{
    "id": "532a98c9-f26a-4500-9053-01deec12e881",
    "log": "Error while accessing datastore;Please retry later",
    "orgId": "orgname",
    "status": "FAILED",
    "type": "ENABLE_MINT"
}

By Investigating /opt/apigee/var/log/edge-management-server/logs/system.log, I found the below errors : 

2018-11-27 20:23:08,282  Apigee-Main-23 INFO  MINT.SERVICE - OrgSetupFacadeImpl.setup() : Completed :com.apigee.mint.setup.SeedBillingDocumentNumbersStep
2018-11-27 20:23:08,282  Apigee-Main-23 INFO  c.a.m.a.AsyncProcessor - AsyncProcessor$1.pre() : listener pre [] 532a98c9-f26a-4500-9053-01deec12e881
2018-11-27 20:23:08,283  Apigee-Main-23 INFO  MINT.SERVICE - OrgSetupFacadeImpl.setup() : Executing :com.apigee.mint.setup.EnableNotificationStep
2018-11-27 20:23:08,348  Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.searchForRecordsWithCriteriaList() :  Couldn't search for specific criteria  
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:26) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:745) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:727) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:132) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getIndexedSlices(KeyspaceServiceImpl.java:749) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:148) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery$1.doInKeyspace(IndexedSlicesQuery.java:139) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:90) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.IndexedSlicesQuery.execute(IndexedSlicesQuery.java:138) ~[hector-core-2.0-0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:417) [datastore-service-1.0.0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) [datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) [mint-1.170821.01-SNAPSHOT.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20464) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result$get_indexed_slices_resultStandardScheme.read(Cassandra.java:20422) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$get_indexed_slices_result.read(Cassandra.java:20348) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
	at org.apache.cassandra.thrift.Cassandra$Client.recv_get_indexed_slices(Cassandra.java:881) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$Client.get_indexed_slices(Cassandra.java:865) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at me.prettyprint.cassandra.service.KeyspaceServiceImpl$19.execute(KeyspaceServiceImpl.java:733) ~[hector-core-2.0-0.jar:na]
	... 25 common frames omitted
2018-11-27 20:23:08,350  Apigee-Main-23 ERROR c.a.m.a.AsyncOrgSetup - AsyncOrgSetup.onError() : Error processing offlinejob com.apigee.mint.model.Job@55b618a8[orgId=orgname,status=RUNNING,type=ENABLE_MINT,log=,id=532a98c9-f26a-4500-9053-01deec12e881,created=Tue Nov 27 20:23:07 UTC 2018,createdBy=<null>,updated=Tue Nov 27 20:23:07 UTC 2018,updatedBy=<null>]
com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithCriteriaList(CassandraClient.java:464) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.datastore.client.CassandraClient.searchForRecordsWithSearchMap(CassandraClient.java:373) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.findAllNotificationEmailTemplateItemsForOrgId(NotificationCassandraDataStore.java:204) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.findAllNotificationEmailTemplateItemsForOrgId(NotificationFacade.java:64) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.resource.NotificationResourceImpl.getNotificationEmailTemplateItems(NotificationResourceImpl.java:140) ~[notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.setDefaultEmailTemplates(EnableNotificationStep.java:137) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.EnableNotificationStep.execute(EnableNotificationStep.java:104) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.OrgSetupFacadeImpl.setup(OrgSetupFacadeImpl.java:59) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.call(AsyncOrgSetup.java:76) ~[mint-1.170821.01-SNAPSHOT.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
2018-11-27 20:23:08,352  Apigee-Main-23 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist
2018-11-27 20:23:08,390  Apigee-Main-23 ERROR DATASTORE.CASSANDRA - CassandraClient.insert() :  @@@@@@@@@@@@@@ CREATING OPERATION FAILED !!!!!!
me.prettyprint.hector.api.exceptions.HUnavailableException: [192.168.56.103(192.168.56.103):9160] : May not be enough replicas present to handle consistency level.
	at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:65) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:102) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69) ~[hector-core-2.0-0.jar:na]
	at com.apigee.datastore.client.CassandraClient.insert(CassandraClient.java:144) ~[datastore-service-1.0.0.jar:na]
	at com.apigee.notification.nosql.NotificationCassandraDataStore.insertNotificationServiceItem(NotificationCassandraDataStore.java:116) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.addNotificationEmailMessage(NotificationFacade.java:147) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.notification.service.NotificationFacade.sendNotificationEmailMessage(NotificationFacade.java:131) [notification-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.util.EmailUtil.scheduleEmail(EmailUtil.java:95) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.setup.NotifyOnCompletionStep.execute(NotifyOnCompletionStep.java:36) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncProcessor.notifyJobStatus(AsyncProcessor.java:88) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncProcessor.updateAndNotify(AsyncProcessor.java:75) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.mint.asyncjob.AsyncOrgSetup.onError(AsyncOrgSetup.java:62) [mint-1.170821.01-SNAPSHOT.jar:na]
	at com.apigee.threadpool.CallbackTask.done(CallbackTask.java:44) [threadpool-1.0.0.jar:na]
	at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.setException(FutureTask.java:251) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:271) [na:1.8.0_131]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.cassandra.thrift.UnavailableException: null
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28411) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result$batch_mutate_resultStandardScheme.read(Cassandra.java:28388) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:28322) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) ~[libthrift-0.9.1.jar:0.9.1]
	at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1078) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1064) ~[cassandra-thrift-2.1.2.jar:2.1.2]
	at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:268) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:265) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104) ~[hector-core-2.0-0.jar:na]
	at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:241) ~[hector-core-2.0-0.jar:na]
	... 23 common frames omitted
2018-11-27 20:23:34,426  qtp1200546947-91 ERROR ORGANIZATIONS - OrganizationPropertiesInfo.getPropertiesForOrganization() : Exception occurred when updating OrganizationPropertiesMap entry for org default@@@org_emf. Ignoring the exception = Organization : default@@@org_emf does not exist


Could anyone advise what is the root cause and how to overcome? 

@anilsagar 
Comment
Add comment Show 1
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Shawky Foda · Feb 04, 2019 at 05:27 AM 0
Link

Steps to Resolve issues due to installing monitization in AIO :

$ cd /opt/apigee/apigee-cassandra-2.1.16-0.0.1123/bin
$ ./cqlsh 192.168.56.103 9042
cqlsh>SELECT * FROM system.schema_keyspaces ;

You will notice that the replication factor = 3 for mint , taurus , notification keyspaces ....

keyspace_name  | durable_writes | strategy_class  | strategy_options
------------------------+----------------+------------------------------------------------------+----------------------------
auth | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"1"}
notification | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"3"}
cache | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"1"}
taurus | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"3"}
devconnect | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"1"}
system_traces | True | org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"2"}
…….
keyvaluemap | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"1"}
mint | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":“3"}
counter | True | org.apache.cassandra.locator.NetworkTopologyStrategy | {"dc-1":"1"}
cqlsh> alter keyspace mint WITH REPLICATION = {'class': 'NetworkTopologyStrategy' , 'dc-1':'1' };
cqlsh> alter keyspacetaurus WITH REPLICATION = {'class': 'NetworkTopologyStrategy' , 'dc-1':'1' };
cqlsh> alter keyspace notification WITH REPLICATION = {'class': 'NetworkTopologyStrategy' , 'dc-1':'1' };

Close

0 Answers

  • Sort: 

Follow this Question

Answers Answers and Comments

69 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Enabling monetization fails 1 Answer

How to configure monetization datastores after adding new region (cassandra-cluster) (4.16.09) 1 Answer

Enable monitization 0 Answers

Using external billing system with monetization. 2 Answers

Monetization setup failed 0 Answers

  • Products
    • Edge - APIs
    • Insights - Big Data
    • Plans
  • Developers
    • Overview
    • Documentation
  • Resources
    • Overview
    • Blog
    • Apigee Institute
    • Academy
    • Documentation
  • Company
    • Overview
    • Press
    • Customers
    • Partners
    • Team
    • Events
    • Careers
    • Contact Us
  • Support
    • Support Overview
    • Documentation
    • Status
    • Edge Support Portal
    • Privacy Policy
    • Terms & Conditions
© 2021 Apigee Corp. All rights reserved. - Apigee Community Terms of Use - Powered by AnswerHub
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Post an idea
  • Spaces
  • Product Announcements
  • General
  • Edge/API Management
  • Developer Portal (Drupal-based)
  • Developer Portal (Integrated)
  • API Design
  • APIM on Istio
  • Extensions
  • Business of APIs
  • Academy/Certification
  • Adapter for Envoy
  • Analytics
  • Events
  • Hybrid
  • Integration (AWS, PCF, Etc.)
  • Microgateway
  • Monetization
  • Private Cloud Deployment
  • 日本語コミュニティ
  • Insights
  • IoT Apigee Link
  • BaaS/Usergrid
  • BaaS Transition/Migration
  • Apigee-127
  • New Customers
  • Explore
  • Topics
  • Questions
  • Articles
  • Ideas
  • Badges