edge-microgateway sidecar container failed at down...

jimlin · 08-15-2019 11:38 PM

I've deployed a working 2-node apigee on-prem installation[1] of version 4.19.01.

Now I'm trying to deploy edge-microgateway[2] into our kubernetes cluster.

The edge-microgateway container sidecar is running however its logs shows there is an error of downloading config.

Here is the complete logs

Log Location: [ /opt/apigee/logs/edgemicro.log ]
cd /opt/apigee && export EDGEMICRO_DECORATOR=1 && export EDGEMICRO_LOCAL_PROXY=1 && edgemicro start -o ikala-cc -e prod -k 39f21d4459c53c5f0ca89a26965b6e7d65e7072ea99767c12be93a5dc276f13c -s 09245a4ef64cf79f851991ade9d8c6ad2d11a1eeb6f5edcef8bff7fabe2bfd1f -r 8000 -d /opt/apigee/plugins -a edgemicro_HELLOWORLD -v 1 -b / -t http://localhost:8081 &
current nodejs version is v8.15.1
current edgemicro version is 3.0.2
config download from http://192.168.0.11:9001/edgemicro/bootstrap/organization/ikala-cc/environment/prod returned
jwt_public_key download from http://192.168.0.11:9001/edgemicro-auth/publicKey returned
products download from http://192.168.0.11:9001/edgemicro-auth/products returned
error downloading config, please check bootstrap configuration Error: Internal Server Error
    at _loadStatus (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:457:12)
    at Request._callback (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:299:17)
    at Request.self.callback (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1161:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)
failed to retieve config from gateway. continuing, will try cached copy..
Error: Internal Server Error
    at _loadStatus (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:457:12)
    at Request._callback (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:299:17)
    at Request.self.callback (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1161:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)
cache configuration /opt/apigee/.edgemicro/ikala-cc-prod-cache-config.yaml does not exist. exiting.

How do I fix this issue?

Documents

1. https://docs.apigee.com/private-cloud/v4.18.05/installation-topologies#standaloneinstallation2host

2. https://docs.apigee.com/api-platform/integrations/kubernetes/k8s-sidecar#using_manual_sidecar_inject...

jimlin

I also try to verify the installation by using the instruction described in the doc[1]

edgemicro verify -o ikala-cc -e prod -k $MY_KEY -s $MY_SEC

The same error is returned which is "Internal Server Error".

2019-08-16T08:11:02.750Z [27110] [microgateway edgemicro] current nodejs version is v12.8.1
2019-08-16T08:11:02.753Z [27110] [microgateway edgemicro] current edgemicro version is 3.0.5
2019-08-16T08:11:03.670Z [27110] [microgateway-config network] jwt_public_key download from http://192.168.0.11:9001/edgemicro-auth/publicKey returned
2019-08-16T08:11:03.673Z [27110] [microgateway-config network] config download from http://192.168.0.11:9001/edgemicro/bootstrap/organization/ikala-cc/environment/prod returned
2019-08-16T08:11:03.718Z [27110] [microgateway-config network] products download from http://192.168.0.11:9001/edgemicro-auth/products returned
2019-08-16T08:11:03.719Z [27110] [microgateway-config network] error downloading config, please check bootstrap configuration Error: Internal Server Error
    at _loadStatus (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:459:12)
    at Request._callback (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:301:17)
    at Request.self.callback (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:185:22)
    at Request.emit (events.js:203:13)
    at Request.<anonymous> (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:203:13)
    at IncomingMessage.<anonymous> (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:291:20)
    at IncomingMessage.emit (events.js:208:15)
    at endReadableNT (_stream_readable.js:1168:12)
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:121
  Object.keys(config).forEach(function (key) {
         ^


TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at IO.save (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:121:10)
    at IO.saveSync (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:105:8)
    at Object.save (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/index.js:44:25)
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/cli/lib/verify.js:231:16
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:139:20
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:339:20
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:726:13
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:52:16
    at done (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:241:17)
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:44:16
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:723:17
    at /home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/async/lib/async.js:167:37
    at _loadStatus (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:459:9)
    at Request._callback (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:301:17)
    at Request.self.callback (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:185:22)
    at Request.emit (events.js:203:13)
    at Request.<anonymous> (/home/jim/.nvm/versions/node/v12.8.1/lib/node_modules/edgemicro/node_modules/microgateway-config/node_modules/request/request.js:1161:10)<br>

It looks like there is something wrong with the "/edgemicro" or "edgemicro-auth" API.

Documents

1. https://docs.apigee.com/api-platform/microgateway/3.0.x/setting-and-configuring-edge-microgateway#pa...

deboraelkin

@jimlin, a few things to check:

Did you complete the prerequisites listed in https://docs.apigee.com/api-platform/integrations/kubernetes/k8s-prerequisites?
Are you using the manual or automatic sidecar injection?
1. If manual, did you run edgemicro private configure with the required --mgmt_url and --api_base_path parameters?
2. If automatic, did you run ./install/kubernetes/webhook-edgemicro-patch.sh with the required --mgmt_url and --api_base_path parameters?
Does the kubernetes deployment where the sidecar is injected have outbound connectivity to the URL specified in --mgmt_url?

jimlin

Q1 Did I complete the prerequisites?

A1 Yes.

Q2 Using the manual or automatic sidecar injection?

A2 Manual.

If manual, did I run edgemicro private configure with the required --mgmt_url and --api_base_path parameters?

Here is the command I run, and it successfully returns key and secret information.

edgemicro private configure -m http://$MGMT_IP:8080 -r http://$ROUTER_IP:9001/ -v default -o ikala-cc -e prod -u $ORG_ADMIN -p $ORG_ADMIN_PASS

Q3 Does the kubernetes deployment where the sidecar is injected have outbound connectivity to the URL specified in --mgmt_url?

A3 Yes.

I also run this command to verify, and it returns message saying "error downloading config, please check bootstrap configuration Error: Internal Server Error".

edgemicro verify -o ikala-cc -e prod -k $THE_KEY -s $THE_SEC

jimlin

The root cause seems to be edgemciro version.

After I remove the installed micro gateway proxy and then re-install edgemciro with different version, it works now.

Here is my steps:

1. Remove "edgemicro-internal" api proxy from the management ui.

2. Remove "edgemicro-auth" api proxy from the management ui.

3. Reinstall edgemicro@3.0.3 (3.0.5 is not work)

4. edgemicro private configure

5. edgemicro veirfy (3.0.5 is not work)

edge-microgateway sidecar container failed at downloading config