Apigee Hybrid First Time Install "GoogleTokenService halting the boot up process"

[tiny-dancer]

First time install of apigee hybrid.  All seems well except the runtime pod, what was missed as part of install?

Error in deploying the deployment : GoogleTokenService halting the boot up process

{"level":"SEVERE","thread":"main","mdc":{},"className":"com.apigee.kernel.MicroKernel","method":"deployAll","severity":"SEVERE","message":"Error in deploying the deployment : GoogleTokenService halting the boot up process","formattedDate":"2021-07-22T21:38:48.389Z","logger":"KERNEL","exceptionStackTrace":"com.apigee.kernel.exceptions.spi.UncheckedException{ code \u003d googletoken.LoadDefaultCredentialFailure, message \u003d Failed to load application default Google credential, associated contexts \u003d [Deployment {name\u003dGoogleTokenService, type\u003dservice, deploymentPath\u003d/opt/apigee/apigee-runtime/lib/infra/services/googletoken-1.0.0.jar}]}\nCaused by: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.\n\tat com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)\n\tat com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:120)\n\tat com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:92)\n\tat com.apigee.googletoken.GoogleIamCredentials.getDefaultCredential(GoogleIamCredentials.java:20)\n\tat com.apigee.googletoken.GoogleTokenServiceImpl.start(GoogleTokenServiceImpl.java:63)\n\tat com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:173)\n\tat com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:73)\n\tat com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:275)\n\tat com.apigee.kernel.MicroKernel.start(MicroKernel.java:156)\n\tat com.apigee.kernel.MicroKernel.main(MicroKernel.java:112)\n"}

[ElangoD]

It shows wrong in the Service Account and key associated with that.

can u check Service Account and respective roles binding to Service Account

[tiny-dancer]

@ElangoD do you mind confirming what service account to be checked?  This is a fresh deploy from apigeectl apply and passed the apigeectl dry run checks (which led to fixing some gcp service account role iam).  When looking at the deployed YAML for the apigee-runtime pod there does not appear to be any configuration to set GOOGLE_APPLICATION_CREDENTIALS, is this expected?

  containers:
    - name: apigee-runtime
      image: 'gcr.io/apigee-release/hybrid/apigee-runtime:1.5.1'
      ports:
        - name: https-8443
          containerPort: 8443
          protocol: TCP
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: APIGEE_DPCOLOR
          value: '151'
        - name: APIGEE_ORG
          value: eval-nonprod-us
        - name: APIGEE_ENV
          value: poc-azure
        - name: APIGEE_REGION
          value: us-central1
        - name: CONTRACT_ENCRYPT_KEY_PATH
          value: /etc/encryption/plainTextDEK
        - name: EVAL_NONPROD_US_KMS_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: eval-nonp-poc-azure-8fafdbb-encryption-keys
              key: kmsEncryptionKey
        - name: EVAL_NONPROD_US_KVM_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: eval-nonp-poc-azure-8fafdbb-encryption-keys
              key: kvmEncryptionKey
        - name: EVAL_NONPROD_US_POC_AZURE_KVM_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: eval-nonp-poc-azure-8fafdbb-encryption-keys
              key: envKvmEncryptionKey
        - name: EVAL_NONPROD_US_POC_AZURE_CACHE_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: eval-nonp-poc-azure-8fafdbb-encryption-keys
              key: cacheEncryptionKey
        - name: AX_HASH_SALT
          valueFrom:
            secretKeyRef:
              name: eval-nonp-c22fc7a-ax-salt
              key: salt

[tiny-dancer]

overrides.yaml

envs:
  - name: poc-azure
    serviceAccountPaths:
      synchronizer: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
      udca: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

mart:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

connectAgent:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
  # Same account used for mart and connectAgent

metrics:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

watcher:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

logger:
  enabled: true # set to "false" for GKE.
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

[tiny-dancer]

Note i observed envs[].serviceAccountPaths.runtime was missing from the example although unfortunately no luck on that resolving start up issue:

envs:
  - name: poc-azure
    serviceAccountPaths:
      synchronizer: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
      udca: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
      runtime: ./service-accounts/eval-nonprod-us-apigee-non-prod.json <--

mart:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

connectAgent:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
  # Same account used for mart and connectAgent

metrics:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

watcher:
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

logger:
  enabled: true # set to "false" for GKE.
  serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json

[Former Community Member · Accepted solution]

Can you please use Apigee hybrid 1.5.2, which was launched yesterday? It should fix this error (and not require the runtime SA).

[tiny-dancer]

thank you @Former Community Member , this resolved the issue.

a couple extra notes for downloading 1.5.2, re-running init + apply.

1) when the apigee-runtime container _is not_ running, the below still returns as:

$APIGEECTL_HOME/apigeectl check-ready -f overrides/overrides.yaml

Parsing file: config/values.yaml

Parsing file: overrides/overrides.yaml

All containers are ready.

2) all containers successfully updated to 1.5.2, _except_ the apigee-runtime container (investigating why).  however the updated apigeectl did correctly assign the `GOOGLE_APPLICATION_CREDENTIALS`