Apigee Hybrid Overrides Split Strategy and Configuration Management

greatdevaks · ‎07-26-2023

This article is co-authored 🤝 by @ashwinknaik, @sampriyadarshi, and @greatdevaks ‌🚀

Introduction

For installing the Apigee Hybrid Runtime, a user must create a configuration overrides file. This file lets the user override the default configuration values as needed, configure Apigee Environments, reference TLS Certificates and Service Account Keys, assign Kubernetes Node Pools to specific Apigee Hybrid Runtime Components, etc.

The complete list of configurable properties can be found in the Configuration Property Reference documentation. The reference documentation only lists the properties which can be modified; if one tries to modify a property that's not in the reference documentation, the change is either ignored or could result in an error when the overrides file is referenced during apigeectl apply. It should be noted that not all the properties are editable.

Apigee Hybrid (through use of apigeectl) maintains its default component configuration in the file <apigee_hybrid_root_dir>/config/values.yaml. The overrides file(s) follow the same YAML structure as that of the values.yaml file. When a configuration is applied to a cluster using apigeectl, the overrides are merged with the defaults (values.yaml) to create the complete Kubernetes cluster configuration; this approach is very similar (yet different) to the Helm Values Files approach.

There are multiple setups/settings for Apigee Hybrid Runtime Component and Configuration Management that can be achieved by tweaking the overrides files, based on the requirements. For some enterprises/organizations, it is a good practice, from a maintenance perspective for handling large Apigee Hybrid ecosystems (when ownership of different components is to be defined), to split the main overrides file into smaller self-contained overrides files (explained later in the article) so that it becomes easier to manage the Apigee Hybrid Runtime Components; also, component specific configurations (like if Apigee Environments are to be sized differently) can be applied and maintained independently. Many users are interested in having different resource configurations for different Apigee Environments, and that is fully achievable by maintaining different self-contained overrides files.

Note: When splitting an overrides file into smaller maintainable component specific overrides files, some practices need to be followed, as explained in the article, in order to adhere to the way apigeectl handles resource provisioning and maintenance. The main focus of this article is to make Apigee Hybrid users aware of the Apigee Hybrid Overrides Split Strategy and Configuration Management for large scale operations.

A sample Apigee Hybrid overrides file (overrides.yaml) is shown below for reference.

# overrides.yaml
gcp:
  region: ANALYTICS_REGION
  projectID: GCP_PROJECT_ID
  workloadIdentityEnabled: true

k8sCluster:
  name: CLUSTER_NAME
  region: CLUSTER_LOCATION # Must be the closest Google Cloud region to your cluster.
org: ORG_NAME

instanceID: "UNIQUE_INSTANCE_IDENTIFIER"

cassandra:
  hostNetwork: false
    # false for all GKE installations.
    # See Multi-region deployment: Prerequisites
  replicaCount: 3
    # Use 1 for demo installations and multiples of 3 for production.
    # See Configure Cassandra for production for guidelines.
  backup:
    enabled: true
    # Set to true for initial installation.
    # This triggers apigeectl to create the apigee-cassandra-backup Kubernetes service account.
    # See Cassandra backup overview for instructions on using cassandra.backup.

virtualhosts:
- name: ENVIRONMENT_GROUP_NAME
  selector:
    app: apigee-ingressgateway
    ingress_name: INGRESS_NAME
  sslCertPath: ./certs/CERT_NAME.pem
  sslKeyPath: ./certs/KEY_NAME.key

ao:
  args:
    disableIstioConfigInAPIServer: true

ingressGateways:
- name: INGRESS_NAME # maximum 17 characters.
  replicaCountMin: 2
  replicaCountMax: 10
  svcAnnotations:  # optional. If you are on AKS, see Known issue #260772383
    SVC_ANNOTATIONS_KEY: SVC_ANNOTATIONS_VALUE
  svcLoadBalancerIP: SVC_LOAD_BALANCER_IP  # optional

envs:
- name: ENVIRONMENT_NAME

logger:
  enabled: false # Set to false for all GKE installations.

Information on how to find more such sample overrides files can be found at the official documentation.

Overrides Split Strategy

When splitting an overrides file into multiple smaller and manageable overrides files, it is important to hold configuration specific to some components in respective independent files. Also, the order in which the overrides files are applied/maintained matters and should always be respected; more details on the same are discussed later in the article.

Shown below is the description of different files that can be created and maintained, along with the commands to apply/maintain them, in order. It is recommended to have some standard naming convention followed to name the overrides files.

Table: Overrides Split Description

Overrides File	Description (purpose of the Overrides File)	Fields/Properties to include in Overrides File (for reference purposes; some are optional) Notes: More fields could be applicable; check the Pro Tip below this table Top-level properties that are required should be included in each overrides file in order to stick with consistent behaviour and avoid issues
common.yaml	For cluster initialization (to be supplied with the apigeectl init command)	namespace => Include this if changing the namespace where Apigee Components are going to get installed; default and recommended namespace is apigee cassandra.auth.secret => The name of the file stored in a Kubernetes secret that contains the Cassandra users and passwords; default is none redis.auth.secret => The name of a file stored in a Kubernetes secret that contains the TLS certificate and private key; default is none
cassandra.yaml	For Cassandra Datastore configuration Note: Similar thing can be done for Redis as well by creating redis.yaml file and placing the redis configuration along with supporting properties (which are very similar to the ones stated for Cassandra)	cassandra => The Cassandra configuration imagePullSecrets => Kubernetes Secret name and details for pulling the Container Images; if getting used namespace => If a custom namespace is getting used for hosting the Apigee Hybrid Runtime Components httpProxy => If any configuration for HTTP Forward Proxy server is to be used nodeSelector => Apigee Runtime and Apigee Data details for Node Selector
org.yaml	For Apigee Organization-scoped resources like connect-agent, mart, watcher, etc. Note: Configuration for ingressGateways also goes in org.yaml overrides file and not the env-group.yaml file	Configuration for components like mart, udca, connectAgent, watcher, namespace, imagePullSecrets, cassandra, httpProxy, nodeSelector, ingressGateways, etc.
test-env.yaml	For Apigee Environment-scoped resources Note: Single file per Apigee Environment could be maintained; if the number of Apigee Environments is high, the Apigee Environments which should have same configuration for Apigee Environment-scoped resources (like runtime, synchronizer, udca, etc.) can be put in the same overrides file	Configuration for namespace, nodeSelector, httpProxy, envs, etc., and Apigee Environment level components like runtime, synchronizer, udca, etc. (in case the resources or properties of these components are to be set differently for different Apigee Environments) watcher.serviceAccountPath should also be specified in each such <name_prefix>-env.yaml Note: Depending on the Overrides Split approach followed for Apigee Environment(s), usage of --env flag or --all-envs flag could be made during apigeectl apply; more on this is explained in an example later
telemetry.yaml	Apigee Telemetry configuration for components like apigee-logger and apigee-metrics	Configuration for components like logger, metrics, namespace, imagePullSecrets, httpProxy, nodeSelector, etc.
env-group.yaml	VirtualHosts configuration	Mainly the configuration for the virtualhosts component

Pro Tip: In order to understand what all fields should be included in an overrides file, run the --dry-run=client command against the select component, like against --env, and then check the corresponding template file by going to <apigee_hybrid_root_dir>/templates/3_apigee_environments.yaml; templates for each category of Apigee Hybrid Runtime Components like Cassandra/Datastore, Organization, Telemetry, etc. are part of the <apigee_hybrid_root_dir>/templates directory. The template files can be explored in order to understand the resources that can get created using the templates and also the fields that are accepted. It is recommended to have the required top-level properties and the other important properties like nodeSelector, imagePullSecrets, httpProxy, etc. carefully evaluated for presence as not including the same could lead to issues and conflicts.

The below-mentioned commands can be used for reference to apply (have the --dry-run done first in order to avoid installation issues) the configurations through multiple overrides files; the order is important.

# Setup system components and perform cluster initialization.
$ apigeectl init -f common.yaml --dry-run=client
$ apigeectl init -f common.yaml

# Setup Cassandra Datastore.
$ apigeectl apply -f cassandra.yaml --datastore --dry-run=client
$ apigeectl apply -f cassandra.yaml --datastore

# Setup Redis Datastore.
$ apigeectl apply -f redis.yaml --redis --dry-run=client
$ apigeectl apply -f redis.yaml --redis

# Setup Apigee Org-scoped components.
$ apigeectl apply -f org.yaml --org <org_name> --dry-run=client
$ apigeectl apply -f org.yaml --org <org_name>

# Setup Runtime level components per Apigee Environment. Run this for each Apigee Environment with its respective YAML file and its mention in the --env flag, if all the Apigee Environments have their dedicated independent overrides files.
$ apigeectl apply -f test-env.yaml --env=test --dry-run=client
$ apigeectl apply -f test-env.yaml --env=test
# The command below can be used if there are multiple Apigee Environments configured in the same overrides file.
$ apigeectl apply -f combined-envs.yaml --all-envs --dry-run=client
$ apigeectl apply -f combined-envs.yaml --all-envs
# Depending on the Overrides Split Strategy and configuration, a mix of both the above-mentioned `--env` and `--all-envs` commands can be used if there are both kinds of overrides files available.
# Note: `--env` can also be used to do a targeted apply for a specific Apigee Environment that is part of a combined overrides file (but this approach doesn't work at scale, nor with automation).

# Setup Telemetry (logger, metrics)
$ apigeectl apply -f telemetry.yaml --telemetry --dry-run=client
$ apigeectl apply -f telemetry.yaml --telemetry

# Setup Ingress Virtual Hosts
$ apigeectl apply -f env-group.yaml --settings=virtualhosts --dry-run=client
$ apigeectl apply -f env-group.yaml --settings=virtualhosts

Example Overrides Split Strategy

Different Kubernetes Resources (CPU/RAM/Replicas) for Different Apigee Environment-specific Runtime Components

One of the most common use-cases for Overrides Split Strategy is to handle Kubernetes Resources like CPU, RAM, HPA configuration, etc. differently for different Apigee Environments. Consider a scenario where there are a total of 6 Apigee Environments being used, named, say, env-01, env-02, ..., env-06. Out of these 6 Apigee Environments, let's assume that one of the Apigee Environments (env-04) hosts API Proxies which serve larger TPS (Transactions Per Second), hence requiring more resources (CPU/RAM) and replicas. Let's assume that env-06 is not to be deployed yet but its configuration, which should be the same as env-04, still needs to be defined. And finally, let's assume that the remaining 4 Apigee Environments have similar resource and replica requirements.

In such a case, two overrides files can be created (talking from Apigee Environment perspective and leaving the configuration for the other components as per the strategy discussed in the previous section), say, light-envs.yaml and heavy-envs.yaml, as shown below.

light-envs.yaml - having configuration for all the Apigee Environments except env-04 and env-06.

# light-envs.yaml
gcp:
 projectID: xxxxxxxx
org: xxxxxxxx

nodeSelector:
 requiredForScheduling: true
 apigeeRuntime:
   key: "cloud.google.com/gke-nodepool"
   value: "apigee-runtime"
 apigeeData:
   key: "cloud.google.com/gke-nodepool"
   value: "apigee-data"

envs:
 - name: env-00
   serviceAccountPaths:
     synchronizer: ./service-accounts/xxxxxxxx-apigee-synchronizer.json
     udca: ./service-accounts/xxxxxxxx-apigee-udca.json
     runtime: ./service-accounts/xxxxxxxx-apigee-runtime.json
 … # More such configuration, except `env-04` and `env-06`.
 - name: env-05
   serviceAccountPaths:
     synchronizer: ./service-accounts/xxxxxxxx-apigee-synchronizer.json
     udca: ./service-accounts/xxxxxxxx-apigee-udca.json
     runtime: ./service-accounts/xxxxxxxx-apigee-runtime.json
      
synchronizer:
 replicaCountMin: 2

runtime:
 replicaCountMin: 1
 replicaCountMax: 3
 resources:
   requests:
     cpu: 50m
     memory: 100Mi

watcher:
 serviceAccountPath: ./service-accounts/xxxxxxxx-apigee-watcher.json

heavy-envs.yaml - having configuration for only env-04 and env-06 Apigee Environments.

# heavy-envs.yaml
gcp:
 projectID: xxxxxxxx
org: xxxxxxxx

nodeSelector:
 requiredForScheduling: true
 apigeeRuntime:
   key: "cloud.google.com/gke-nodepool"
   value: "apigee-runtime"
 apigeeData:
   key: "cloud.google.com/gke-nodepool"
   value: "apigee-data"

envs:
 - name: env-04
   serviceAccountPaths:
     synchronizer: ./service-accounts/xxxxxxxx-apigee-synchronizer.json
     udca: ./service-accounts/xxxxxxxx-apigee-udca.json
     runtime: ./service-accounts/xxxxxxxx-apigee-runtime.json
 - name: env-06 # To be deployed in future.
   serviceAccountPaths:
     synchronizer: ./service-accounts/xxxxxxxx-apigee-synchronizer.json
     udca: ./service-accounts/xxxxxxxx-apigee-udca.json
     runtime: ./service-accounts/xxxxxxxx-apigee-runtime.json
      
synchronizer:
 replicaCountMin: 4

runtime:
 replicaCountMin: 4
 replicaCountMax: 10
 resources:
   requests:
     cpu: 100m
     memory: 200Mi

watcher:
 serviceAccountPath: ./service-accounts/xxxxxxxx-apigee-watcher.json

The above-mentioned overrides files can then be applied, as shown below, to achieve the desired output.

# Applying configuration for only `env-04` using heavy-envs.yaml.
$ apigeectl apply -f heavy-envs.yaml --env=env-04 --dry-run=client
$ apigeectl apply -f heavy-envs.yaml --env=env-04

# Applying light-envs.yaml.
$ apigeectl apply -f light-envs.yaml --all-envs --dry-run=client
$ apigeectl apply -f light-envs.yaml --all-envs

Overrides File Versioning

Use SCM (Source Code Management) for managing overrides files' versions.
Use external secrets management to template secrets.
Use overrides files' versions for rollbacks, if needed.
Use staged rollouts to higher environments.
Store/Maintain apigeectl version information together with the overrides files in SCM.

Recommendations and Noteworthy Points

The gcp and org fields/properties, at least, will need to be included in each overrides file. k8sCluster and instanceID along with other top-level properties can be included on a needs basis (--dry-run=client will error out in case any field is required but missing).
It is not necessary to split the overrides file, nor it is necessary, if splitting, to split in the manner shown in this article. One of the key takeaways from this article is that a user (apigeectl administrator) should be aware of the fields/properties that affect a component (Organization, Environment, Datastore, etc.) and in what sense.

Conclusion

The article tried to highlight some of the strategies to manage the Apigee Hybrid Overrides Files at scale using the Overrides Split Strategy. There are more parts to this series of articles focused on managing Apigee Hybrid Overrides Files and the same will be published soon.

Hope you found this article helpful. Stay tuned 📺