Apigee Hybrid Security Best Practices: Encrypting Sensitive Helm Values

greatdevaks · ‎02-08-2024

This article has been co-authored 🤝 by @devashishpatil and @greatdevaks ‌🚀‌🚀🚀

Intent

This article is intended for users who are concerned about Apigee Hybrid Helm Charts storing sensitive values in raw/plaintext form in Helm Values/Overrides file(s).

Disclaimer

This article talks about making use of some popular open-source tools for handling file content encryption and storing sensitive configuration values for Apigee Hybrid Helm Charts. The implementation shown and the tools used are neither official Google products nor are officially supported by Google.

Background

Starting version v1.10.x, support for provisioning and managing Apigee Hybrid Runtime Plane Components using Helm Charts was introduced in Preview. As of the time of writing this article, v1.11.x is the latest version of Apigee Hybrid and Apigee Hybrid Helm Charts have achieved general availability (GA); reference: Apigee Hybrid Runtime Setup with Helm Charts for v1.11.x.

There are eight (8) Helm Charts, as of the time of writing this article, for deploying Apigee Hybrid Runtime Plane Components like Apigee Operator, Apigee Datastore, Apigee Telemetry, Apigee Redis, Apigee Ingress Manager, Apigee Organization, Apigee Environment, and Apigee Virtual Host (Environment Group). It is important to note that for Apigee Environment and Apigee Virtual Host (Environment Group) Helm Charts, if runtime components for multiple Apigee Environments and Apigee Environment Groups are to be provisioned and managed, dedicated Helm Releases are required for each Apigee Environment and Apigee Environment Group, respectively.

Each Apigee Hybrid Helm Chart is arranged in a similar manner, as shown below (only the important/relevant files and directories have been highlighted for brevity).

# Note: <component> can be one of `operator`, `datastore`, `telemetry`, `redis`, `ingress-manager`, `org`, `env,` or `virtualhost`.
# Check the official documentation for more details: https://cloud.google.com/apigee/docs/hybrid/v1.11/helm-install-helm-charts.

apigee-<component>
  |----templates/
  |----.helmignore
  |----Chart.yaml
  ...
  |----values.yaml

As shown above, the values.yaml file is different for each Apigee Hybrid Helm Chart as the configuration requirements of each Apigee Hybrid Runtime Plane Component may differ. A values.yaml file contains a default set of configuration that is used to generate the Kubernetes resources corresponding to a Helm Chart.

Apigee Hybrid Helm Charts use defaults (defined as part of values.yaml) for most of the configuration settings. However, there are a few configuration settings that do not have defaults but are still required for the creation of desired Kubernetes resources. Moreover, users generally need control to override the configuration settings that are defined through defaults. Helm allows configurations to be overridden for Helm Charts, as explained in the official Helm documentation.

By convention, configuration overrides for Apigee Hybrid Helm Charts are written in a file named overrides.yaml. The overrides.yaml file is typically stored in the directory which stores all the Apigee Hybrid Helm Charts ($APIGEE_HELM_CHARTS_HOME directory). For more information on how to create overrides.yaml file, check out the public documentation. For information about the configurations which can be tweaked through overrides.yaml, check out the Apigee Hybrid Configuration Property Reference documentation.

Apigee Hybrid Runtime Plane Components which are deployed through Helm Charts have three ways of ingesting configuration (defined through values.yaml and overrides.yaml files):

(a) Kubernetes Secrets: A Kubernetes Secret can be created, wherever supported by Apigee Hybrid, and the name of the Kubernetes Secret can be referenced as part of the configuration settings in the overrides.yaml file. For example, Cassandra authentication settings can be configured to be ingested using Kubernetes Secrets.

(b) External Secret Manager: This option is pretty new and is currently only applicable for storing Cassandra authentication settings in Hashicorp Vault. As of the time of writing this article, the feature of using an External Secret Manager (Hashicorp Vault) for storing Cassandra authentication settings is in Preview. The Kubernetes Secrets Store CSI Driver API is used to create a SecretProviderClass object which can interact with Hashicorp Vault and retrieve secrets. The Cassandra authentication settings can then be supplied to the desired Apigee Hybrid Helm Charts (apigee-operator and apigee-datastore) using overrides.yaml where a reference to the SecretProviderClass object is made.

(c) Raw YAML Configurations: This option involves defining configuration settings that don’t make use of any Secret Manager and that are available in plaintext (mostly). Some configuration settings, like the ones for Apigee Hybrid Data Encryption Keys, are first Base64-encoded and then specified in overrides.yaml file.

Problem Statement

Users who have strong security requirements may not want some of their configuration settings to be committed to Source / Version Control System(s) in an unencrypted format, even though if the configuration settings are Base64-encoded (like in case of Apigee Hybrid Data Encryption Keys). It is important to note the difference between encoding and encryption. Encoding transforms the data from one format to another so that it can be understood and consumed by different systems. Anyone knowing the encoding algorithm used for encoding the data can decode the data and view its raw form. On the other hand, encryption makes the data unreadable and scrambled for an unauthorized user. Only legitimate user(s) possessing the decryption key(s) can decrypt and read the data.

It could even be a case where users having strong security requirements don’t want to commit encrypted data to their Source / Version Control System(s) but reference sensitive data from some External Secret Manager during installation/upgrade of an Apigee Hybrid Helm Chart.

This article focuses on explaining the ways and practices that can be used for securely handling sensitive Helm Values for Apigee Hybrid Helm Charts.

High-Level Approaches

Since we are dealing with Helm Charts, the approaches defined in this article will make use of helm-secrets plugin. helm-secrets is a Helm plugin that enables decryption of encrypted Helm Values file(s) on the fly during installation/upgrade of a Helm Chart. helm-secrets also provides a functionality to dynamically evaluate secret references inside Helm Template(s) i.e. if some configuration settings defined in the Helm Values file(s) references to some secrets stored in an External Secret Manager like Hashicorp Vault or GCP Secret Manager, helm-secrets can evaluate such secret references inside Helm Template(s) and inject the desired values to the corresponding Helm Chart.

The following two approaches are going to be discussed in this article for handling sensitive Helm Values for Apigee Hybrid Helm Charts:

(a) Approach 1: Overrides File Encryption using helm-secrets and SOPS (Secrets OPerationS)

(b) Approach 2: Dynamic Secret Reference Evaluation using helm-secrets and Helmfile vals

Prerequisites

Before we begin discussing about the above-mentioned two approaches, a couple of prerequisites need to be completed, as stated below:

(a) GCP Project and Apigee Hybrid Organization (Control Plane) should have been set up; reference: official documentation

(b) Helm v3.10+

(c) Steps until installation of Apigee Hybrid Helm Charts should have been completed; v1.10.x reference and v1.11.x reference

(d) helm-secrets Helm Plugin should have been installed; the approaches shown in this article use v4.5.1 of helm-secrets Helm Plugin

(e) [Required for Approach 1] SOPS (Secrets OPerationS)

(f) Hashicorp Vault (described below)

Note: This article covers provisioning of Apigee Hybrid Runtime Plane v1.10.4 (latest patch of 1.10.x as of the time of writing this document).

Approach 1: Overrides File Encryption using helm-secrets and SOPS (Secrets OPerationS)

As stated in the prerequisites section above, it is expected that steps until installation of Apigee Hybrid Helm Charts have been completed.

This approach makes use of helm-secrets with SOPS (Secrets OPerationS) for encrypting file content of overrides.yaml (Helm Values). The helm-secrets plugin does not handle the encryption/decryption operations itself and instead delegates the same to SOPS.

SOPS is an open-source editor for handling encryption/decryption of files. It started as a project at Mozilla and got donated to CNCF as a Sandbox Project in 2023. SOPS supports YAML, JSON, ENV, INI, and BINARY file formats and supports encryption of file content with keys and methods from AWS KMS, GCP KMS, Azure Key Vault, age, GPG (PGP), Hashicorp Vault, and more vendors.

As compared to other encryption tools available, SOPS provides the below-mentioned benefits because of which it was chosen:

Supports GPG and Key Management/Encryption Services from various Cloud Providers like AWS, Google, Azure, Hashicorp, etc.
For structured data (YAML, JSON, etc.), whole-file and partial-file encryption are possible i.e. one can either encrypt all the values keeping the keys intact or can encrypt selective keys; for unstructured data, the whole file can be encrypted
Native integration with helm-secrets Helm Plugin that makes it a suitable candidate for Helm Release management

To read about other alternatives like git-crypt, Transcrypt, BlackBox, etc., check out this amazing external public blog.

Configuring SOPS and Hashicorp Vault

Hashicorp Vault is a Secret Manager and can be used for Encryption and Key Management use-cases as well. More can be read about the Hashicorp Vault Project at its official website.

Hashicorp Vault has been chosen because of it being a common choice for many users and its seamless integration with SOPS for Encryption and Key Management. However, GCP KMS, AWS KMS, Azure Key Vault, etc. could have been used as well. Check out the SOPS documentation for supported Encryption and Key Management Providers.

Hashicorp Vault Configuration

The demo prepared for this article uses a Helm-powered Hashicorp Vault setup in Kubernetes. The official Hashicorp Vault guide can be used for instructions related to Helm-powered installation. Latest image tag for hashicorp/vault container image can be found here.

Using a Helm-powered installation is not necessary and instead any existing reachable/accessible Hashicorp Vault setup can be used. Helm will need access to Hashicorp Vault as well as helm-secrets and SOPS while deploying Apigee Hybrid Helm Charts.

Once the Hashicorp Vault application is up and running, the status of the same can be validated by running the following commands.

kubectl exec -it -n vault vault-0 -- vault status

The output of the above-mentioned commands should look like the one shown below.

Figure: Hashicorp Vault Status Validation

If Hashicorp Vault is being used for the first time, initialization and unsealing will be required. Check out the official Kubernetes Deployment Guide for Hashicorp Vault for more details on how to get Hashicorp Vault initialized.

Once Hashicorp Vault initialization is complete, the same can be configured to provide Cryptography as a Service to SOPS. Hashicorp Vault can be configured with a Transit Secrets Engine to handle encryption functionality for SOPS. Transit Secrets Engine provides the flexibility to encrypt data while storing the encrypted data at the primary data store/location. Through this functionality, content of overrides.yaml, say, can be encrypted and committed to a Source / Version Control System.

The below-mentioned commands can be used to create a Transit Secrets Engine quickly with a valid key.

vault secrets enable -path=sops transit
vault write sops/keys/<keyname> type=rsa-4096

# To view the created key, the following command can be used.
vault read sops/keys/<keyname>

Once the key for SOPS has been created, the same can be viewed, as shown below.

Figure: Hashicorp Vault Key for SOPS

SOPS and Hashicorp Vault Integration

In order for SOPS to use the Hashicorp Vault Key for encryption purposes, SOPS needs to be configured.

Since SOPS is a file editor for handling file encryption, its configuration requires specifying reference to the encryption key while creating a file. It should be noted that the file which has to be encrypted (be it fully or partially) needs to be created using the sops command line utility, as shown below.

Note: Before using Hashicorp Vault with SOPS, reachability/accessibility of Hashicorp Vault needs to be established. If Hashicorp Vault Kubernetes Service is not exposed as Load Balancer Service Type (for Kubernetes-based demo purposes), say, kubectl port-forward should also do for this demo.

# Make Hashicorp Vault reachable/accesible to the environment running the sops command line utility.
kubectl port-forward vault-0 -n vault 8200:8200 # Hashicorp Vault should be accessible via http://127.0.0.1:8200 after port forwarding is done.

# It should be noted that in order for the sops command line utilty to work with Hashicorp Vault, VAULT_ADDR and VAULT_TOKEN Environment Variables need to be available.
export VAULT_ADDR=http://127.0.0.1:8200 # Address to reach the Hashicorp Vault server.
export VAULT_TOKEN=<vault-token-for-access> # Vault Token for accessing Hashicorp Vault.

# `--hc-vault-transit` flag helps specify the Hashicorp Vault Key's URI for encryption pruposes.
sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/<keyname> <filename>.yaml

# The above command should open an editor where the desired YAML configuration that needs to be encrypted can be provided.

Once the above-mentioned commands are executed, a file named <filename>.yaml will be created with encrypted structured YAML data (values will be encrypted, keys will not).

In order to understand how SOPS works under the hood, check out the SOPS Encryption Protocol guide.

Deploying Apigee Hybrid Helm Charts With Encrypted Values

Preparing The Encrypted Overrides Using SOPS

This article makes use of the above-shown sops command which generates a file with encrypted structured data.

From Apigee Hybrid Helm Charts’s perspective, an interesting use-case can be picked for exercising encryption. Default Encryption Keys, as of the latest Apigee Hybrid Runtime Plane version v1.11.x, are stored in values.yaml file and can be modified (with certain restrictions, as stated in the official documentation) during the initial roll-out using overrides.yaml.

In order to override the values for the above-referenced default encryption keys, the following block needs to be added to the overrides.yaml file.

# Check out the public documentation for constraints related to default encryption key modification and key creation: https://cloud.google.com/apigee/docs/hybrid/v1.11/key-encryption#how-to-create-an-encoded-key

defaults:
  org:
    kmsEncryptionKey: <base64-encoded-key>
    kvmEncryptionKey: <base64-encoded-key>
  env:
    kvmEncryptionKey: <base64-encoded-key>
    cacheEncryptionKey: <base64-encoded-key>

Since the data encryption keys could be considered as sensitive information, instead of adding the above-mentioned configuration block to overrides.yaml, a separate file, say secrets.yaml, can be created using the sops command line utility for storing the data encryption keys in encrypted format. Follow the process outlined below to generate an encrypted file containing the data encryption keys for Apigee Hybrid.

sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/<keyname> secrets.yaml

# An editor will open (default is vim) where the data encryption key YAML configuration can be put. Put the configuration and save the file.
defaults:
  org:
    kmsEncryptionKey: <base64-encoded-key>
    kvmEncryptionKey: <base64-encoded-key>
  env:
    kvmEncryptionKey: <base64-encoded-key>
    cacheEncryptionKey: <base64-encoded-key>

Once the secrets.yaml file has been saved, if an attempt is made to view the contents of the file using regular text editors or utilities (vim, VS Code, cat, etc.), encrypted content will be shown instead of the raw/plaintext data. The same is shown below for the configuration used for this demo.

Figure: Encrypted secrets.yaml File Containing Data Encryption Keys for Apigee Hybrid

Any file encrypted through SOPS will also have some metadata added for encryption/decryption purposes to it, as shown above under the sops key. The metadata in the figure shown reflects the Hashicorp Vault’s configuration and Message Authentication Code (MAC).

If the original unencrypted content of the file is to be viewed or edited, there are a couple of ways to do so, as stated below.

(a) Use the sops command line utility by specifying the filename, as shown below.

# The original unencrypted data can be modified using the command shown.
sops secrets.yaml

(b) SOPS VSCode Extension can be used which allows viewing and editing of the original unencrypted data by just opening the encrypted file. Once the encrypted file is opened in VSCode, the user gets redirected to a temporary decrypted version of the file where modifications can be done, as shown below. Changing focus to any other file removes the temporary decrypted file for security purposes.

Figure: Decrypted secrets.yaml File Containing Viewed Using SOPS VSCode Extension

More information on SOPS can be found in an another good external public blog.

Deploying Apigee Hybrid Helm Charts

Once the encrypted secrets.yaml overrides file is available, helm-secrets Helm Plugin can be leveraged for performing deployments for Apigee Hybrid Helm Charts.

A thing to note here is that usage of helm-secrets Helm Plugin requires a slight modification to the original helm install / helm upgrade commands. The helm install / helm upgrade commands translate to helm secrets install / helm secrets upgrade commands.

The sub-sections below show an actual deployment of Apigee Hybrid Helm Charts using the helm-secrets Helm Plugin, encrypted secrets.yaml overrides which is prepared through the use of SOPS and Hashicorp Vault, and the official reference guide for Apigee Hybrid Helm Charts (the demo uses v1.10.x specific guide; refer to v1.11.x guide here).

Do notice the Helm command used for deployments, as shown below. It makes use of helm secrets install / helm secrets upgrade, overrides.yaml (raw non-sensitive configuration), and secrets.yaml (sensitive encrypted configuration).

helm secrets upgrade <release-name> <chart-reference>/ \
    --install \
    --namespace <namespace> \
    --atomic \
    -f overrides.yaml \
    -f secrets.yaml

Step 1: Apigee Operator Installation

Step 2: Apigee Datastore Installation

Step 3: Apigee Telemetry Installation

Step 4: Apigee Redis Installation

Step 5: Apigee Ingress Manager Installation

Step 6: Apigee Organization Installation

Step 7: Installation of Apigee Environment(s)

Step 8: Installation of Apigee Virtual Host(s) / Environment Group(s)

Note: For installing the Apigee Virtual Host Helm Chart, certificate creation step will be required.

Exposing Ingress and Deploying API Proxy for Validation

The steps outlined in the official documentation can be used for reference purposes.

The figure below shows that Apigee Ingress functionality worked fine for the demo.

Figure: Apigee Ingress Validation

Shown below is a mock API Proxy which was deployed for validation purposes.

Figure: Mock API Proxy Deployment

Figure: Mock API Proxy Validation

Do More With SOPS

Configuring Multiple Encryption Providers and .sops.yaml

SOPS is highly configurable and supports configuring encryption keys from multiple providers at the same time for redundancy purposes. For example, GPG can be used for redundancy purposes so that if Hashicorp Vault is not accessible (due to any reason), a PGP Key can be used for encryption/decryption purposes. More details on configuring SOPS can be found at the official SOPS documentation and at an external public blog. The linked external public blog also talks about configuring .sops.yaml file for configuring encryption providers and encryption behavior (which files to encrypt etc.).

Partial Encryption With SOPS

SOPS allows for selective/partial encryption as well where only values corresponding to some specific keys are encrypted. This way, for Apigee Hybrid Helm Charts, one can maintain a single overrides.yaml file with both non-sensitive and sensitive content within it.

A good reference for the selective/partial encryption can be found in this great external public blog.

Handling CI/CD Pipelines With SOPS

(c) Configuring CI/CD Pipelines to use helm-secrets, SOPS, and Encryption Provider is pretty straightforward. The encrypted files can be committed to the Source / Version Control System(s) and the CI/CD Pipeline(s) can reference the Encryption Provider configuration through CI/CD Pipeline Variables/Secrets (for Hashicorp Vault, $VAULT_ADDR and $VAULT_TOKEN are needed). The CI/CD Pipeline Runner(s) should have the binaries for helm-secrets and SOPS available.

Data Key Rotation With SOPS

(d) Key rotation is pretty straightforward with SOPS, as shown below.

sops -r -i secrets.yaml

The above-shown command will re-encrypt the secrets.yaml file with a new data key using the chosen encryption provider.

Other Important Considerations

(a) Though SOPS can allow encryption of sensitive values for Helm (when used with helm-secrets), anyone having access to kubeconfig can trigger the below-mentioned command to see the raw values used for Helm.

helm get values -n apigee <apigee-org-release-name>

This requires locking down Kubernetes RBAC in order to restrict people from accessing sensitive information. More information on the same can be found in the Restricting Access to In-Cluster Helm Values section.

Approach 2: Dynamic Secret Reference Evaluation using helm-secrets and Helmfile vals

As discussed in the above-mentioned Approach 1: Overrides File Encryption using helm-secrets and SOPS (Secrets OPerationS), one of the ways to handle sensitive Helm Values is to use encryption technique and store encrypted files in the codebase. However, there is another approach which helm-secrets Helm Plugin has to offer. The approach is to integrate with an external Secret Manager like Hashicorp Vault and dynamically (on-the-fly) evaluate and inject the values of secrets against the desired configuration keys in the values.yaml or overrides.yaml files. The same can be achieved by using Helmfile vals along with helm-secrets. This approach doesn’t require any encrypted version of files to be committed to Source / Version Control System(s) and instead there is a dynamic evaluation of reference to the secrets stored in the chosen external Secret Manager.

In order to configure Helmfile vals, either HELM_SECRETS_BACKEND=vals needs to be exported as an Environment Variable or -b vals CLI option needs to be passed to helm-secrets Helm Plugin (like helm secrets -b vals). It should also be noted that values.yaml or overrides.yaml files will need to be present inside the directory of each Helm Chart for the Helm Charts using Helmfile vals.

Preparing Hashicorp Vault for Storing Key-Value Pairs

Storing secrets in Hashicorp Vault is straightforward. A KV Secrets Engine can be created for storing generic Key-Value pairs in the physical storage of Hashicorp Vault. In order to enable versioning for keys, KV Version 2 should be used. More details can be found in the official documentation for Hashicorp Vault.

The command shown below can be used to create a KV Secrets Engine Key-Value Store for storing secrets which need to be referenced using Helmfile vals.

vault secrets enable -path=<custom-path-to-kv-store> kv-v2
# For example, `vault secrets enable -path=vals-secrets kv-v2` will create a Key-Value Store named `vals-secrets` of type `kv-v2` i.e. KV Secrets Engine Version 2.

Once the KV Secrets Engine Key-Value Store has been created, secrets can be stored in the same. The secrets can then be referenced in the values.yaml or overrides.yaml files for dynamically injecting the values of secrets during Helm Release deployment.

For demonstration purposes, the setup from Approach 1: Overrides Files Encryption using helm-secrets and SOPS (Secrets OPerationS) can be extended and modified. The SOPS approach will be dropped and instead Helmfile vals approach will be taken for referencing the sensitive values.

As of this demonstration, two secrets will be created in Hashicorp Vault and the same will be referenced in the overrides.yaml files for the respective Helm Charts (as stated above, Helmfile vals when used with helm-secrets require the files containing Helm Values to be present in the Helm Chart’s directory).

The command shown below can be used for creating the desired secrets in Hashicorp Vault.

vault kv put secret/apigee/runtime \
    encryption="YWxvdmVhcGlzMTIzNDU2Nw==" \
    replicas=5
# Though `replicas` may not look like a sensitive value, the same has been shown for demonstrating how the overall concept of dynamic reference to secrets works.

After the execution of the above-mentioned command, the Hashicorp Vault UI can also be checked for confirming if the secrets were created properly, as shown below.

Figure: Hashicorp Vault KV Secrets Engine Key-Value Store

Preparing Helm Values

Once the desired secrets have been successfully created in Hashicorp Vault, the same can be referenced inside the values.yaml or overrides.yaml files, as shown below. It should be noted that the configuration shown below belongs to an overrides.yaml file which is present inside the apigee-env Helm Chart. Likewise the desired Helm Values files can be created in the respective Helm Charts.

Figure: Referencing Hashicorp Vault Secrets in Helm Values using Helmfile vals

The above-shown expression ref+vault://secret/apigee/runtime?proto=http#/replicas should be observed carefully. This kind of expression is required by Helmfile vals to dynamically reference secrets from supported external Secret Managers.

The expression can be understood by breaking it down, as shown.

ref+<external-secret-manager-backend>://<kv-store-name>/<path-to-secrets>?proto=http#/<secret-name>

More details around Helmfile vals’s expression syntax can be found at the official documentation.

Deploying Helm Charts with helm-secrets and Helmfile vals

In order to deploy (either a fresh installation or an upgrade) a Helm Release (for example, an Apigee Environment), the below-mentioned command can be used.

# If HELM_SECRETS_BACKEND=vals is not set as an Environment Variable, `-b vals` CI Option can be used, as shown.
helm secret -b vals template <helm-release-name> <helm-chart-reference> \
    --namespace <namespace> \
    --atomic \
    -f <helm-chart-reference>/overrides.yaml

helm secret -b vals upgrade <helm-release-name> <helm-chart-reference> \
    --install \
    --namespace <namespace> \
    --atomic \
    -f <helm-chart-reference>/overrides.yaml

The figure shown below uses the above-mentioned command to deploy the Apigee Hybrid apigee-org Helm Chart using helm-secrets and Helmfile vals.

Figure: apigee-org Helm Chart’s Deployment using helm-secrets and Helmfile vals

The following overrides.yaml configuration was used for the deployment of the apigee-org Helm Chart.

Figure: apigee-org Helm Chart’s Helm Values

Deployment of the apigee-org Helm Chart was intentional to show that if one is even moving/migrating from SOPS to Helmfile vals, there won’t be any difference if the above-mentioned steps have been followed properly.

To further confirm if the Helm Chart’s Release resulted in introducing any changes/differences, helm-diff Helm Plugin can be used, as shown below. The diff is taken between the revisions 3 and 2 of the deployed apigee-org Helm Chart Release named aks-apg-hybrid-playground-10.

Figure: helm-diff output for apigee-org Helm Chart

Likewise, to demonstrate the impact of ref+vault://secret/apigee/runtime?proto=http#/replica configuration, as shown below, the Apigee Hybrid apigee-env Helm Chart was deployed.

runtime: 
  replicaCountMax: "ref+vault://secret/apigee/runtime?proto=http#/replicas"

As expected, the maxReplicas property for Apigee Environment Runtime got changed from 4 (default value) to 5 (one referenced from Hashicorp Vault using the above-shown configuration). The same can be checked using helm-diff Helm Plugin, as shown below.

helm diff -n apigee revision apigee-env-dev-01 10 9 # diff between revision 10 and 9.

Figure: helm-diff output for apigee-env Helm Chart

With the above-shown demonstrations, the capabilities of helm-secrets and Helmfile vals were highlighted.

Restricting Access to In-Cluster Helm Values

So far, the sections covered how sensitive Helm Values could be handled when committing Helm Charts to Source / Version Control System(s). Though it is a good security measure to not have sensitive configurations pushed to the Source / Version Control System(s), just not committing the sensitive configurations doesn’t provide full protection/security either.

Sensitive configurations such as passwords and secrets which are passed (through Helm Values) while creating a Helm Release are visible to users with excessive permissions (through kubeconfig and Kubernetes RBAC). The helm get values command can be used to access such sensitive information, as shown below. For many Apigee Hybrid users who are keen on enforcing strong security, this might not be an acceptable behavior and can also pose security risks.

Figure: Accessing Sensitive Information Through In-Cluster Helm Values

The Helm Values are available as part of a Helm Release’s Kubernetes Secrets that Helm uses under the hood to persist metadata for a given Helm Release.

Kubernetes RBAC could be used to deal with scenarios requiring security lock-down from the perspective of accessing the In-Cluster Helm Values (using helm get values command). Kubernetes Role(s) and Role Binding(s) could be created to enforce security and control unintended access to Helm Values. More information for configuring least-privilege access to Kubernetes Secrets can be found at the official Kubernetes documentation.

A point to keep in mind here is that Kubernetes RBAC does not have an explicit deny policy or deny rules (the RBAC Permissions are purely additive) i.e. if there are users who need to access Kubernetes Secrets in a cluster, the Roles used for those users should be updated to fine-grained / restricted ones which only allow access to the intended resources.

If a user not having the required RBAC Roles/Permissions tries to access Helm Values without, the following error should be observed.

Figure: Accessing Sensitive Information Through In-Cluster Helm Values Without Proper RBAC

If a set of users are required to have access to Kubernetes Secrets corresponding to Helm Releases, the following Role and Role Binding should be created.

The following file (role.yaml) can be created for defining the Kubernetes Role which allows accessing Kubernetes Secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: apigee
  name: secret-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

The following file (rolebinding.yaml) can be created for defining the Kubernetes Role Binding for the users who should be allowed to access Kubernetes Secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: read-secrets
 namespace: apigee
subjects:
 - kind: User
   name: testuser@devpat.joonix.net # "name" is case sensitive
   apiGroup: rbac.authorization.k8s.io
roleRef:
 # "roleRef" specifies the binding to a Role / ClusterRole
 kind: Role # this must be Role or ClusterRole
 name: secret-reader # this must match the name of the desired Role/ClusterRole
 apiGroup: rbac.authorization.k8s.io

The following commands can be used to apply the above-shown Kubernetes RBAC configuration.

kubectl apply -f role.yaml
kubectl apply -f rolebinding.yaml

For advanced / fine-grained control over access to Kubernetes Secrets and sensitive information within a Kubernetes Cluster, Open Policy Agent or Kyverno can be explored.

Hashicorp Vault User Lock-Down

In order to restrict a user’s access to specific Hashicorp Vault Secrets Engine or Paths inside a Secret Store, the following external public blogs could be referenced:

(a) Access Specific Path Inside a KV Secret

(b) Restricting Access in Hashicorp Vault

(c) Create Hashicorp Vault Tokens for Authentication

Using the information mentioned in the above-referenced external public blogs, a user with restricted access can be created for CI/CD purposes and a Hashicorp Vault Token can be generated for use with helm-secrets.

FAQs

(a) Can SOPS be applied to apigeectl-based deployment of Apigee Hybrid Runtime Plane Components?

Answer: SOPS can be used for apigeectl-based deployment of Apigee Hybrid Runtime Plane Components but that would not require the use of helm-secrets Helm Plugin. Instead, one would need to write their own wrapper on top of SOPS and apigeectl which decrypts the encrypted values.yaml or overrides.yaml files before triggering apigeectl. One such example can be found for using SOPS with Helm without helm-secrets Helm Plugin. This should be good to get an idea of how to work with SOPS and the same can be applied for apigeectl-based deployments.

(b) Can SOPS and Helmfile vals be used together when using helm-secrets?

Answer: helm-secrets doesn’t allow setting SOPS and Helmfile vals as the backends at the same time i.e. only one of them can be used at a time. However, there is a trick to deal with such a use-case, if such a capability is really needed. The trick is to use SOPS as the backend for helm-secrets and pass additional Helm Values that are prepared (dynamic reference resolution) using the Helmfile vals CLI, as shown below.

vals eval -f secrets.yaml # This command prepares secrets.yaml file by resolving ref+vault://secret/apigee/runtime?proto=http#/encryptionKey, say. The secrets.yaml file then be passed to Helm Release Deployment.

helm secrets upgrade aks-apg-hybrid-playground-10 apigee-org/ \
    --install \
    --namespace apigee \
    --atomic \
    -f overrides.yaml \
    -f secrets.yaml

Another approach is to use SOPS as a standalone CLI tool and use helm-secrets with Helmfile vals.

Conclusion

This article covered in detail how to handle sensitive Helm Values for Apigee Hybrid Helm Charts. The following two approaches were explained along with step-by-step guides to exercise them.