First time install of apigee hybrid. All seems well except the runtime pod, what was missed as part of install?
Error in deploying the deployment : GoogleTokenService halting the boot up process
{"level":"SEVERE","thread":"main","mdc":{},"className":"com.apigee.kernel.MicroKernel","method":"deployAll","severity":"SEVERE","message":"Error in deploying the deployment : GoogleTokenService halting the boot up process","formattedDate":"2021-07-22T21:38:48.389Z","logger":"KERNEL","exceptionStackTrace":"com.apigee.kernel.exceptions.spi.UncheckedException{ code \u003d googletoken.LoadDefaultCredentialFailure, message \u003d Failed to load application default Google credential, associated contexts \u003d [Deployment {name\u003dGoogleTokenService, type\u003dservice, deploymentPath\u003d/opt/apigee/apigee-runtime/lib/infra/services/googletoken-1.0.0.jar}]}\nCaused by: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.\n\tat com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)\n\tat com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:120)\n\tat com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:92)\n\tat com.apigee.googletoken.GoogleIamCredentials.getDefaultCredential(GoogleIamCredentials.java:20)\n\tat com.apigee.googletoken.GoogleTokenServiceImpl.start(GoogleTokenServiceImpl.java:63)\n\tat com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:173)\n\tat com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:73)\n\tat com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:275)\n\tat com.apigee.kernel.MicroKernel.start(MicroKernel.java:156)\n\tat com.apigee.kernel.MicroKernel.main(MicroKernel.java:112)\n"}
Solved! Go to Solution.
Can you please use Apigee hybrid 1.5.2, which was launched yesterday? It should fix this error (and not require the runtime SA).
It shows wrong in the Service Account and key associated with that.
can u check Service Account and respective roles binding to Service Account
@ElangoD do you mind confirming what service account to be checked? This is a fresh deploy from apigeectl apply and passed the apigeectl dry run checks (which led to fixing some gcp service account role iam). When looking at the deployed YAML for the apigee-runtime pod there does not appear to be any configuration to set GOOGLE_APPLICATION_CREDENTIALS, is this expected?
containers:
- name: apigee-runtime
image: 'gcr.io/apigee-release/hybrid/apigee-runtime:1.5.1'
ports:
- name: https-8443
containerPort: 8443
protocol: TCP
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: APIGEE_DPCOLOR
value: '151'
- name: APIGEE_ORG
value: eval-nonprod-us
- name: APIGEE_ENV
value: poc-azure
- name: APIGEE_REGION
value: us-central1
- name: CONTRACT_ENCRYPT_KEY_PATH
value: /etc/encryption/plainTextDEK
- name: EVAL_NONPROD_US_KMS_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: eval-nonp-poc-azure-8fafdbb-encryption-keys
key: kmsEncryptionKey
- name: EVAL_NONPROD_US_KVM_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: eval-nonp-poc-azure-8fafdbb-encryption-keys
key: kvmEncryptionKey
- name: EVAL_NONPROD_US_POC_AZURE_KVM_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: eval-nonp-poc-azure-8fafdbb-encryption-keys
key: envKvmEncryptionKey
- name: EVAL_NONPROD_US_POC_AZURE_CACHE_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: eval-nonp-poc-azure-8fafdbb-encryption-keys
key: cacheEncryptionKey
- name: AX_HASH_SALT
valueFrom:
secretKeyRef:
name: eval-nonp-c22fc7a-ax-salt
key: salt
overrides.yaml
envs:
- name: poc-azure
serviceAccountPaths:
synchronizer: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
udca: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
mart:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
connectAgent:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
# Same account used for mart and connectAgent
metrics:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
watcher:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
logger:
enabled: true # set to "false" for GKE.
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
Note i observed envs[].serviceAccountPaths.runtime was missing from the example although unfortunately no luck on that resolving start up issue:
envs:
- name: poc-azure
serviceAccountPaths:
synchronizer: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
udca: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
runtime: ./service-accounts/eval-nonprod-us-apigee-non-prod.json <--
mart:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
connectAgent:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
# Same account used for mart and connectAgent
metrics:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
watcher:
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
logger:
enabled: true # set to "false" for GKE.
serviceAccountPath: ./service-accounts/eval-nonprod-us-apigee-non-prod.json
Can you please use Apigee hybrid 1.5.2, which was launched yesterday? It should fix this error (and not require the runtime SA).
thank you @Former Community Member , this resolved the issue.
a couple extra notes for downloading 1.5.2, re-running init + apply.
1) when the apigee-runtime container _is not_ running, the below still returns as:
$APIGEECTL_HOME/apigeectl check-ready -f overrides/overrides.yaml
Parsing file: config/values.yaml
Parsing file: overrides/overrides.yaml
All containers are ready.
2) all containers successfully updated to 1.5.2, _except_ the apigee-runtime container (investigating why). however the updated apigeectl did correctly assign the `GOOGLE_APPLICATION_CREDENTIALS`