How to Rotate Service Account Keys in Apigee hybrid environments?

[imesh]

Introduction

The Apigee hybrid runtime components use Google Cloud IAM service accounts for authenticating and authorizing API requests sent to Google Cloud APIs. As you may already know a service account is a special kind of account used by an application or compute workload, such as a Compute Engine virtual machine (VM) instance, a container rather than a person. The list of Service Accounts and their roles which are used by Apigee hybrid runtime containers can be found here.

First of all it is important to note that in Apigee hybrid there are two main options for configuring Service Accounts:

1. Using Workload Identity
2. Using Service Account Keys

Option 1: Using Workload Identity

Workload Identity is a feature in GCP which allows a Kubernetes service account in a GKE cluster to act as an IAM service account. Pods that use the configured Kubernetes service account automatically authenticate as the IAM service account when accessing Google Cloud APIs. (Reference [1] ) As a result, this model allows workloads deployed on a GKE cluster to authenticate and authorize without having to use service account keys. Therefore is you use this approach, it is not required to rotate service account keys.

GKE recommends using this approach for authenticating Google Cloud APIs unless there are no other reasons for using Service Account Keys. Instructions for enabling Workload Identity in Apigee hybrid on GKE can be found here.

Option 2: Using Service Account Keys

If Workload Identity is not an option you could use service account keys for this purpose. In this approach we need to generate a key for each service account and configure that in overrides.yaml file either by specifying the service account key JSON file paths or using Kubernetes secrets. It is important to note that when using service account keys, those should be managed securely without being leaked. Best practices for managing service account keys can be found here.

When using service account keys users may need to revoke them due to security breaches or periodically rotate them to reduce security risk caused by leaked keys.

In this article I will explain how to rotate service account keys in Apigee hybrid deployments without causing a downtime.

Service Accounts and Their Usage

Here, note that the service accounts are used by the following Apigee runtime components:

• Apigee Cassandra Backup: 
  • Allows Cassandra backups to Cloud Storage, as described in Backup and recovery.
• Apigee Logger:
  • Allows logging data collection, as described in Logging. Only required for non-GKE cluster installations.
• Apigee MART:
  • Allows MART service authentication. The Apigee Connect Agent role allows it to communicate securely with the Apigee Connect process, as described in Using Apigee Connect.
• Apigee Connect Agent:
  • Both Apigee MART and Connect Agent use the same service account and role.
• Apigee Metrics:
  • Allows metrics data collection, as described in Metrics collection overview.
• Apigee Runtime:
  • Allows the Apigee hybrid runtime to connect to Google services and custom services on Google Cloud, such as Google Authentication, Google Cloud Trace, and Jaeger.
• Apigee Synchronizer:
  • Allows the synchronizer to download proxy bundles and environment configuration data. Also enables operation of the trace feature.
• Apigee UDCA:
  • Allows the transfer of trace, analytics and deployment status data to the management plane.
• Apigee Watcher:
  • Apigee Watcher pulls virtual hosts related changes for an org from synchronizer and makes necessary changes to configure istio ingress.

Reference:
https://cloud.google.com/apigee/docs/hybrid/v1.6/sa-about#recommended-sas

Out of the above list, it is not required to restart Apigee Cassandra pods because the Cassandra service account is used by the Apigee Cassandra Backup pods generated by the backup cronjob. Hence, it will pick the new service account credential secret on the next execution. The Apigee Logger is orchestrated by a Daemon Set and others are orchestrated by Apigee Deployments/ReplicaSets. Hence, we will be using those entities for executing rolling restarts to Apigee runtime pods.

How to Rotate Service Account Keys in Apigee hybrid

1. Generate a new service account key for each Apigee component service account either via Google Cloud console or gcloud CLI and download the key JSON files to the local machine in which Apigee hybrid installation was executed:
https://cloud.google.com/iam/docs/creating-managing-service-account-keys

2. If service account file paths have been defined in the overrides.yaml file update those accordingly. Otherwise, if secrets have been defined in the overrides.yaml file, recreate those secrets in the Kubernetes cluster:
https://cloud.google.com/apigee/docs/hybrid/v1.6/install-configure-cluster#specify-configuration-ove...

3. If service account JSON file paths are set in the overrides.yaml file execute apigeectl apply command to update the Kubernetes secrets generated for the service account credentials:

cd hybrid-files/
apigeectl apply -f overrides/overrides.yaml

4. Execute the below command for restarting apigee-logger pods:

kubectl -n apigee rollout restart daemonset apigee-logger-apigee-telemetry

The kubectl rollout restart daemonset commandwill restart each apigee-logger-apigee-telemetry pod without causing a downtime.

An example output:

kubectl -n apigee rollout restart daemonset apigee-logger-apigee-telemetry
daemonset.apps/apigee-logger-apigee-telemetry restarted

5. Wait until all apigee-logger pods are restarted:

watch kubectl -n apigee get pods -l app=apigee-logger

6. Restart rest of the Apigee runtime pods managed by Apigee Deployments (short name: ad) by executing below command:

kubectl -n apigee patch ad $(kubectl get ad -n apigee --template '{{range .items}}{{.metadata.name}}{{" "}}{{end}}') -p '{"release":{"replaceWithClone":true}}' --type=merge

This command will list all the Apigee Deployments available in the apigee namespace and rollout restart all of them including apigee-redis-envoy pods similar to the kubectl rollout restart command.

In this command, kubectl get ad -n apigee --template '{{range .items}}{{.metadata.name}}{{" "}}{{end}}' is used for listing the Apigee Deployments and the remaining part is used for patching each Apigee Deployment and setting {"release":{"replaceWithClone":true}}:

7. Wait until all Apigee runtime pods are restarted:

watch kubectl -n apigee get pods

8. Now, verify the health of the Apigee hybrid runtime by sending API requests to API proxies deployed.

Hope this article will help, please take a few minutes to share your feedback in comments. Thanks!

[strebel]

Great stuff! Thanks for walking through the different steps! 

For when I am too lazy I generally just use AHR's `sa-keys-rotate` function: https://github.com/apigee/ahr/wiki/ahr-sa-keys-management#ahr-sa-ctl-sa-keys-rotate

ahr credit to @yuriyl