How to identify which proxy is generating errors?

[#1]

We've occasionally seen important errors in the message-processor system.log, but we haven't been able to figure out how to determine which of our many proxies it is for.

Here's an example log:

2016-07-18 00:01:51,641 org:MyOrganization env:prod Apigee-Main-93 ERROR MESSAGING.FLOW - AsyncExecutionStrategy$AsyncExecutionTask.logException() : Exception caught 
com.apigee.oauth.v2.OauthAdaptorVerificationException: Invalid ApiKey
        at com.apigee.oauth.v2.OAuthServiceImpl.getClientAttributesForApiKey(OAuthServiceImpl.java:519) ~[oauthV2-1.0.0.jar:na]
        at com.apigee.steps.oauth.v2.OAuthStepExecution.execute(OAuthStepExecution.java:418) ~[oauthV2-1.0.0.jar:na]
        at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:132) ~[message-processor-1.0.0.jar:na]
        at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:87) [message-flow-1.0.0.jar:na]
        at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:56) [message-flow-1.0.0.jar:na]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]

This frequent error on our internal-only Apigee Edge indicates that a client is misconfigured and using the wrong OAuth credentials or perhaps a standard API Key. The problem is that it does not indicate what proxy the error is originating from. We have dozens of proxies. It shows the organization and environment but that's all.

This problem isn't limited to the ApiKey error, but includes other important errors that we'd like to efficiently track down.

Any suggestions?

Using OPDK 4.15.07.00

[sudheendras]

Hi @Eric Dahl Did you check the Error Analytics dashboard? The proxy errors section should indicate the API proxy causing this error. Also the Analytics tab under APIs menu will give you a report based on various metrics including Error. You can drill down further based on the API proxy to identify the app/developer causing this issue.

If you need further assistance, raise a ticket & let Apigee Support help you.

[#3]

This helps, but I still would need to correlate the error from the logs with the proxy from this UI. That is, if there's multiple errors occurring for different proxies, how do I know which error is associated with which proxy? The Analytics UI doesn't seem to show details about the errors, other than a count.

Is there a reason why the proxy name can't be included in the error message?

[sudheendras]

You could include a Message Logging policy (either to syslog or file-log) in the fault handling flow with a customized log message to include API proxy & other details. Our logging has gone through some changes in the last couple of releases. I see the API proxy name along with revision # in the error log in our recent release -

2016-07-18 22:28:38,149 org:myorg env:prod api:HelloWorld rev:2 messageid:ip-10-0-0-32.ec2.internal-23099-165-2  Apigee-Main-2 ERROR MESSAGING.FLOW - AbstractAsyncExecutionStrategy$AsyncExecutionTask.logException() : Exception caught

I don't know the reason for not including proxy names in logs... Sorry! May be @archendra can help.

[#5]

As @sudheendra1 pointed out, you can use the message-logging policy to segregate the logs including the customization. As far as message logging is concerned, we had changed our logging capabilities drastically in our recent releases. You should consider upgrading to our latest private cloud release to utilize the recent features, etc.

Thanks,

Archnedra

[#6]

@archendra @sudheendra1 is there a doc or something that outlines the major changes?

We have previously been forced to do a lot of custom stuff to make up for some of the holes in general logging and i would love to be able to drop support for some of that code....

[#7]

Great. That updated log info will be very useful, though we'll have to update our logstash groks.