Microgateway Config Error

Not applicable

I'm getting an error on the generating keys portion of the microgateway private configuration steps. See Below:

generating keys { [Error: cannot POST http://routerURL.int.thomsonreuters.com/edgemicro/credential/organization/gatewayteam/environment/in... (401)] text: undefined }

Router VIP and MGMT Vips are correct. I've also verified that admin email and password is correct. Doesn't matter if I use URLs or IP:Ports I still get the same error. Network connectivity appears to be in tact between the new edgemicro and the existing MGMT and Router servers.

2 38 1,265
38 REPLIES 38

Hi Bryant,

A 401 Unauthorized would suggestion there is an issue with your credentials/role. Can you confirm that the same credentials work for a different Management API call?

e.g. http://docs.apigee.com/management/apis/get/organizations/%7Borg_name%7D/apis/%7Bapi_name%7D

Not applicable

Yes, the credentials work with other management API calls, I've confirmed that. The issue, is that the URL http://int.api.apigarden-test.int.thomsonreuters.com/ is the url of the router, not the management server. When you run the setup command the -r flag says to specify the router IP address. That's the one I provided, it appears the -m portion, which is the management server works fine as it setup the two proxies and made it all the way to the generating keys part.

prabhat
Participant V

Hi Bryant,

Since you are using on-prem installation, you should see edgemicro-internal proxy deployed to your org and env. You can run a trace on that proxy to see where it's failing. Pls make sure that you are working with same environment throughout.

If that does not solve, set environment variable 'DEBUG=*' and run genkeys command again to get more details.

So it's definitely trying to post to the below path:

http://routerServer.int.thomsonreuters.com:9001/edgemicro/credential/organization/gatewayteam/enviro...

But it's receiving a 401 unauthorized. Would the admin account credentials not work for this post for some reason?

david_ryan
Participant V

@Bryant Olson - Did you every get an answer to this? We are having the same issue in our environment. The management sever URL seems fine, as configure creates the 2 api proxies to support edgemicro. Then it tries to hit the router URL and we get a 401. I've tried both debug and trace and not able to gleen any information as it seems the error is originating from the java callout which we have no documentation for.

As a side we do have our routers for this env listening on port 9003 but we front the routers with loadbalancers listening on port 443.

so our -r is https://loadbalancer.api.domain.com not http://loadbalancer.api.domain.com:9003

I'm not sure if this plays into it at all but wondering if your doing something similar.

Not applicable

Hi @David Ryan,

I haven't gotten an answer yet but I was able to get it working on an AIO box but still experience it from time to time. We also use a load balancer with port translation just like you, but I've tried calling the RMP directly as and receive the same result so I don't think it actually matters. With that said I was able to get my AIO box working by restarting the entire stack. I'm thinking that either just restarting the Router/Message Processor or just the management server would do it but I'm yet to test that theory. I wanted to gather a bit more information around this before I brought it to Apigee but I think it's a bug on their end that will probably need some attention. Ultimately we'd like to put this in a docker image and make this self serve if possible, but that as well as having to use the org admin account and password make that a bit of a challenge.

Well, I think I just may have found the issue, but now how to fix it. I had gone through and recycled all my services which didn't make a difference. I then tail'd the message-processor logs and got an error when running the edgemicro private configure command.

Looks like the Java callout is trying to connect to MGMT https://api.enterprise.apigee.com which doesn't seem right. I didn't specify this anywhere in my config. my management server URL it totally different, and again, it's working as the proxies are getting created.

2016-02-05 11:12:00,207 org:myorg env:test api:edgemicro-internal rev:1 policy:Callout vxpid-eampsb03.lmig.com_CUVSE5C6M_RouterProxy-8-340_1 Apigee-Main-6 ERROR EdgeMicoJavaCallout - Credential.authenticateToUserOrg() : **** unable to connect to MGMT https://api.enterprise.apigee.com/v1/o/ets java.net.SocketTimeoutException: connect timed out

Interesting, have you ever been able to successfully configure an edgemicro server? This is interesting as I've been able to do it a handful of times on a cloud machine.

@Martin Nally have you or anyone on your team seen this issue? I feel like there's a bug somewhere in this code.

No I haven't and suspect the reason is our servers are not allowed out the internet, hence the connect timeout to the api.enterprise.apigee.com url. I need to figure out where this is coming from and try to change it to our managment server URL.

David,

Since it's your on-prem environment, when you do 'edgemicro private configur' on CLI, it should update cli/config/default.yaml with MGMT API url for your installation.

If value of prop managementUri is https://api.enterprise.apigee.com then pls try it manually updating it. The default url is for Edge Cloud which does not apply to you.

We have other on-prem customers run it successfully so either it's something specific to your environment or there is a bug during config.

@prabhat - I validated that the cli/config/default.yml has the correct mgment URL so i'm still not sure where it's getting this vaule (https://api.enterprise.apigee.com). Also I did look at the proxy code that was created by the CLI and those mgmt server values are correct as well.

@David Ryan This is really weird. Just to confirm: you are on OPDK 1507 and Microgateway 1.1.1 ? If answer to both is yes then let's schedule some time to debug it. Pls send me an email at prabhatATapigeeDOTcom.

We're running 4.15.07.03 in all of our environments and I'm not seeing it anymore in my cloud environment. If possible I'd like to join this troubleshooting call.

Hey Bryant, Since you are on OPDK, what do you mean by not seeing anymore in your cloud environment?

that is correct 15.07.03 and microgateway 1.1.1. Will shoot you an email now

We have OPDK installed on 5 different stacks, Private Cloud(openstack) and 4 in our data centers (test, dev, qa, and 2 prod). I run an AIO box on openstack and I'm not seeing the behavior there anymore. I haven't had a chance to run the configuration on our data center installs yet.

interestingly - i did all of the "needful" yesterday. 1) updated jar from proxy - restarted mp - still saw issues.

2) updated the settings in the proxy policy thing - didnt restart - still saw issues. 3) this morning i restarted MP only and then saw a DIFFERENT message in the MP logs - then i fixed the policy settings (the FQDN for my Router requires SSL now - and I had it in the PROXY as http) Restarted and everything suddenly worked.

Side note: it appears that the MP setting in the CLI for microgateway has no impact on this particular problem unless its your first install maybe? but even then i dont know.

Not applicable

Very interesting, have you ever successfully configured an edgemicro server?

Not applicable

It looks like heavily upgraded installs will have somehow missed some permissions settings. When tracing on the "edgemicro-internal" proxy.

We are getting a 401 on the java callout in the RESPONSE - in the messageprocessor we see this:

2016-02-15 04:28:01,282 org:<myorg> env:<myenv> Apigee-Main-2 ERROR EdgeMicoJavaCallout - Credential.authenticateToUserOrg() : **** unable to connect to MGMT https://api.enterprise.apigee.com/v1/o/gatewayteam java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_79] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_79] at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_79] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) ~[na:1.7.0_79] at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.7.0_79] at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.7.0_79] at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:275) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371) ~[na:1.7.0_79] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[na:1.7.0_79] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933) ~[na:1.7.0_79] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[na:1.7.0_79] at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301) ~[na:1.7.0_79] at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.7.0_79] at com.apigee.edgemicro.javacallout.Credential.authenticateToUserOrg(Credential.java:166) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.edgemicro.javacallout.Credential.execute(Credential.java:83) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.edgemicro.javacallout.Callout.execute(Callout.java:130) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$ClassLoadWrappedExecution.execute(JavaCalloutStepDefinition.java:164) [javacallout-1.0.0.jar:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution$1.run(JavaCalloutStepDefinition.java:229) [javacallout-1.0.0.jar:na] at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_79] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution.execute(JavaCalloutStepDefinition.java:226) [javacallout-1.0.0.jar:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$CallOutWrapper.execute(JavaCalloutStepDefinition.java:99) [javacallout-1.0.0.jar:na] at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:132) [message-processor-1.0.0.jar:na] at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:87) [message-flow-1.0.0.jar:na] at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:56) [message-flow-1.0.0.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

I will send you a .jar file and see if it fixes this issue.

That didnt work. I have actually already taken a JAR from an AIO to see if this works. It looks like there is a config entry SOMEWHERE that we need to update/fix.

Id love to try to find this so i can resolve this and demo by Wednesday.

The jar didn't work for us either. I can't help but feel that this is a result of something that needs to be changed in a properties file somewhere else as well.

@prabhat - Do you have any other thoughts?

The updated jar didn't work for either of us. Do you have any other ideas?

the updated JAR did work for us - i just revisited it w/ a clear head to make sure I didnt miss anything.

I had to check the config under callout: i had forgotten that we had moved our mgmt to https only recently.

Also - it seems like you need to restart the MP after making the jar change? Not sure why.

I did restart our MP's base on Prabhat's instructions. Can you share exactly how you applied the updated jar? Maybe it's in the process of updating the jar.

yeah... erm.. let me think about how to best do this. Lots of pictures usually breaks the community app...

Update your jar file here:

1982-apigee-issue-2.jpg

make sure these are updated:

1981-apigee-issue-1.jpg

Ah, ok, yea, those values for us have always been correct in our case. Just wondered how you updated the JAR itself. I've done it 2 ways. 1) Replacing the one in the cli under resources and then using configure. 2) Added the new jar as a resource in the proxy code itself.

In either case I restarted our RMP's with no luck. Waiting to hear back from Prabhat on next steps.

Not applicable

going to open a ticket for this.

prabhat
Participant V

Property is there and in your edgemicro-internal proxy, you should see the property in Callout policy. Bug is that it's not picking it up.

Okay I saw that and tried to edit it (ip vs fqdn for our management service...) and it didnt do anything. We are still pointing to the wrong location in this call.

im going to delete and try to recreate the proxy its self maybe?

Not applicable

Benjamin,

Can you grep for "MGMT_URL_PREF" in your log file. and post the result. Thanks.

I grepped the logs on both the message processor server and the micro edge server that Ben has been working on and didn't see MGMT_URL_PREF in any of them. Or were you referring to Ben's traces?

Try to grep for "MGMT" in the message processor logs instead.

sudo grep "MGMT" /opt/apigee4/var/log/apigee/message-processor/logs/system.log

2016-02-16 09:10:10,037 org:internal env:ci-sandbox api:edgemicro-internal rev:1 policy:Callout vxpid-eampsb04.lmig.com_CUWQYJNPC_RouterProxy-10-172176_1 Apigee-Main-7 ERROR EdgeMicoJavaCallout - Credential.authenticateToUserOrg() : **** unable to connect to MGMT https://api.enterprise.apigee.com/v1/o/internal

In our logs it's clear that the java callout is trying to contact https://api.enterprise.apigee.com instead of our on-prem management server URL, which Prabhat points out above there is some bug that is not picking up the correct management server URL.

I also wanted to note @David Ryan that I was able to get this to clear up by restarting ONLY the message processor service and attempting the configuration again. that

You're right, it's there grep just wouldn't grab it because the log had rolled over and was tar'd.

2016-02-12 21:43:21,199 org:micro_aio env:development api:edgemicro-internal rev:3 messageid:micro-gateway-aio.emea1.cis.trcloud_CUWXNTVP6_RouterProxy-6-2_1 policy:Callout Apigee-Main-2 ERROR EdgeMicoJavaCallout - Credential.authenticateToUserOrg() : **** unable to connect to MGMT https://api.enterprise.apigee.com/v1/o/micro_aio java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_79] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_79] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_79] at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_79] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) ~[na:1.7.0_79] at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.7.0_79] at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.7.0_79] at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:275) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371) ~[na:1.7.0_79] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[na:1.7.0_79] at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933) ~[na:1.7.0_79] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[na:1.7.0_79] at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301) ~[na:1.7.0_79] at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) ~[na:1.7.0_79] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.7.0_79] at com.apigee.edgemicro.javacallout.Credential.authenticateToUserOrg(Credential.java:166) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.edgemicro.javacallout.Credential.execute(Credential.java:83) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.edgemicro.javacallout.Callout.execute(Callout.java:130) [edge-micro-javacallout-1.0.0.jar/:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$ClassLoadWrappedExecution.execute(JavaCalloutStepDefinition.java:164) [javacallout-1.0.0.jar:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution$1.run(JavaCalloutStepDefinition.java:229) [javacallout-1.0.0.jar:na] at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_79] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution.execute(JavaCalloutStepDefinition.java:226) [javacallout-1.0.0.jar:na] at com.apigee.steps.javacallout.JavaCalloutStepDefinition$CallOutWrapper.execute(JavaCalloutStepDefinition.java:99) [javacallout-1.0.0.jar:na] at com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:132) [message-processor-1.0.0.jar:na] at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:87) [message-flow-1.0.0.jar:na] at com.apigee.flow.execution.AsyncExecutionStrategy$AsyncExecutionTask.call(AsyncExecutionStrategy.java:56) [message-flow-1.0.0.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]