Message Processor Error log Error Service Cache

Not applicable

Hi everyone,

We find several errors in our message-processor log, the error says ERROR SERVICE.CACHE and generates a NullPointerException, we start using the response cache a few days ago, but we disabled every cache policy and restart the nodes with de router and MP's but the error still continue.

Have you a similar error.

Thanks for your help.

Emilio

The details is:

org:telefonicachile env:prod api:deviceV1 rev:7 messageid:n4-r-mp.cloudtelefonica.local-30640-362-95 policy:verify-api-key Apigee-Main-5 ERROR SERVICES.CACHE - DistributedCacheManagerImpl.getEntry() : DistributedCacheManagerImpl.getEntry() - exception when fetching entry organizations@@@telefonicachile - ignoring it....

java.lang.NullPointerException: null

at com.apigee.cache.CacheMemoryLimiter.get(CacheMemoryLimiter.java:164) ~[cache-1.0.0.jar:na]

at com.apigee.cache.CacheManagerImpl.updateMemoryLimiter(CacheManagerImpl.java:144) ~[cache-1.0.0.jar:na]

at com.apigee.cache.CacheManagerImpl.getEntryFromCache(CacheManagerImpl.java:255) ~[cache-1.0.0.jar:na]

at com.apigee.cache.CacheManagerImpl.getEntry(CacheManagerImpl.java:247) ~[cache-1.0.0.jar:na]

at com.apigee.cache.distributed.DistributedCacheManagerImpl.getEntry(DistributedCacheManagerImpl.java:228) ~[cache-1.0.0.jar:na]

at com.apigee.keymanagement.util.CacheUtil.getFromCache(CacheUtil.java:194) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.dao.nosql.impl.BaseDaoImpl.fetchEntityFromDataStore(BaseDaoImpl.java:439) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.dao.nosql.impl.BaseDaoImpl.getByKey(BaseDaoImpl.java:303) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.dao.nosql.impl.BaseDaoImpl.getByKey(BaseDaoImpl.java:299) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.dao.nosql.impl.OrganizationDaoImpl.getByName(OrganizationDaoImpl.java:53) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.util.ResourceUtil.getOrganization(ResourceUtil.java:295) [keymanagement-1.0.0.jar:na]

at com.apigee.keymanagement.service.OrganizationServiceImpl.getOrganization(OrganizationServiceImpl.java:179) [keymanagement-1.0.0.jar:na]

at com.apigee.developer.service.DeveloperServiceImpl.assertOrganizationExists(DeveloperServiceImpl.java:580) [developer-1.0.0.jar:na]

at com.apigee.developer.service.DeveloperServiceImpl.getDeveloperForIdWithLazyLoadCompanyDetails(DeveloperServiceImpl.java:316) [developer-1.0.0.jar:na]

at com.apigee.oauth.v2.connectors.LocalOAuthServiceConnector.getClientAttributesWithDeveloperEmail(LocalOAuthServiceConnector.java:289) [oauthV2-1.0.0.jar:na]

at com.apigee.oauth.v2.connectors.LocalOAuthServiceConnector.getClientAttributesForApiKey(LocalOAuthServiceConnector.java:202) [oauthV2-1.0.0.jar:na]

at com.apigee.oauth.v2.OAuthServiceImpl.getClientAttributesForApiKey(OAuthServiceImpl.java:519) [oauthV2-1.0.0.jar:na]

at com.apigee.steps.oauth.v2.OAuthStepExecution.execute(OAuthStepExecution.java:466) [oauthV2-1.0.0.jar:na]

at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:132) [message-processor-1.0.0.jar:na]

at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:87) [message-flow-1.0.0.jar:na]

at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:56) [message-flow-1.0.0.jar:na]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_55]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]

Solved Solved
0 5 717
1 ACCEPTED SOLUTION

adas
Participant V

@Emilio Fallau apart from the explicit response cache policy and other cache policies, some policies like the verifyapikey or oauth or access entity policies also try to cache few things so that we do bot make calls to cassandra for the same access token or apikey for each request. This error seems related to that. To dig into the issue, we would need some more details like:

1. on-premise version

2. deployment architecture and topology

3. detailed system logs from the message processor

4. sample apiproxy bundle

I understand most of this is sensitive information that you may not want to share publicly, hence I would suggest you open a support ticket and provide all these details. Someone from Apigee's global support center should be able to help you.

If you are not seeing any functional issues with this, its most probably harmless but I think it has to do with trying to propagate the cache on one message processor to the other message processors for distributed caching. One of the reasons could be that message processors are not able to talk to each other over rpc, so if you have multiple mps it could be a problem. Try doing a telnet from one mp to the other on the rpc port 4526.

View solution in original post

5 REPLIES 5

adas
Participant V

@Emilio Fallau apart from the explicit response cache policy and other cache policies, some policies like the verifyapikey or oauth or access entity policies also try to cache few things so that we do bot make calls to cassandra for the same access token or apikey for each request. This error seems related to that. To dig into the issue, we would need some more details like:

1. on-premise version

2. deployment architecture and topology

3. detailed system logs from the message processor

4. sample apiproxy bundle

I understand most of this is sensitive information that you may not want to share publicly, hence I would suggest you open a support ticket and provide all these details. Someone from Apigee's global support center should be able to help you.

If you are not seeing any functional issues with this, its most probably harmless but I think it has to do with trying to propagate the cache on one message processor to the other message processors for distributed caching. One of the reasons could be that message processors are not able to talk to each other over rpc, so if you have multiple mps it could be a problem. Try doing a telnet from one mp to the other on the rpc port 4526.

Not applicable

Thanks for the quick answer @arghya das. We believe that your suggest is correct, because the error start when in a mp started to show permgen error, in that moment the mp cannot response well the request and in the log shows troubles associated to the rpc in the mp.

Regards

Emilio

Not applicable

When we restart the mp's after a kill -9 for a permgen error, we saw that behavior, then, stop and start de mp's, if the mp down correct, we start again and then sincronize correct the caches

Thanks again @arghya das

mpurwar
Participant IV

@Emilio Fallau Can you please list down the exact steps you performed to get past this issue.

I have already re-started the MPs but still the same issue, also which version of OPDK are you using ?

Thanks

Mudit

Not applicable

@Mudit.P the steps are, we have the permGen issue, we try to stop the MP with the command apigee-services (we have 16.02) but doesn't work. we kill process id for the java task for MP (kill-9)

We check with apigee-services status that the MP process is down(the process indicates dead)

We start the MP with apigee services start-all, and then we execute stop all, we check in the log that says "WebServer INFO o.e.j.u.t.ShutdownThread - ShutdownThread.hook() : shutdown already commenced” and the we start all with apigee services, with that, we can't see the issue for null pointers because the cache isn't sincronized.

Regards

Milo