This article is co-authored 🤝 by @ashwinknaik, @sampriyadarshi, and @greatdevaks 🚀
Problem Statement
As per the Apigee Hybrid Kubernetes Cluster creation recommended practices, a minimum of two Node Pools should be created for a cluster, as stated below.
One Node Pool named, say, apigee-data (a statefulset Node Pool) and another named, say, apigee-runtime (a stateless Node Pool) should be created. More information can be found here.
The apigee-data Node Pool should only run the Cassandra workloads. For the remaining Apigee Runtime Components (non-Cassandra components) like runtime, synchronizer, udca, etc., it is possible to define nodeSelector configuration in the overrides.yaml file so that all the Apigee Runtime Components (non-Cassandra) get scheduled on the apigee-runtime Node Pool. However, for some other additional components like cert-manager and asm (asm is not relevant for users running Apigee Ingress Gateway instead of ASM and who have decommissioned ASM already; Apigee Ingress Gateway feature got introduced in Apigee Hybrid v1.8), specifying nodeSelector configuration may not be straightforward. As a result, sometimes, workloads like cert-manager and asm could get deployed on the apigee-data Node Pool, thereby occupying resources, due to which the scheduling of the Cassandra Pods could become a problem and some Cassandra Pods may not get any free Worker Nodes to get scheduled on.
Resolution
During the Apigee Hybrid Kubernetes Cluster creation, when creating/configuring (preferred time for configuring the below-mentioned properties) the Node Pools (and even after the Node Pools have been created), there is an option to specify the Node Taints. Node Taints allow a Worker Node to repel a set of Pods based on the taint effect and the associated key-value pairs.
Tolerations should be applied to the Pods which should be allowed to be scheduled onto the Worker Nodes with matching taints.
Note: Properties for Taints and Tolerations have been recently added for some of the Apigee Hybrid Runtime Components (some components like Cassandra, Runtime, etc. used to support Taints and Tolerations in Apigee Hybrid v1.7.x+ versions as well) in the Apigee Hybrid v1.10.1 version. It is highly recommended to evaluate such configurations in some lower/Non-Production ecosystem before deploying them to Production.
In order to have only the Cassandra workloads deployed on the apigee-data Node Pool, Node Taints should be applied to the apigee-data Node Pool.
The following command (note: it is just for reference purposes and ideally no configurations should be applied manually to Kubernetes Clusters, especially if using Anthos Clusters, as [a] reconciliation or recreation of cluster components, if happens due to any reason, could result in disaster; [b] applying taints should be considered as part of Infrastructure as Code;) places a taint on Worker Node node1. The taint has a key named workload with value cassandra-datastore, and taint effect NoSchedule. This means that no Pod will be able to get scheduled onto node1 unless it has a matching toleration.
kubectl taint nodes node1 workload=cassandra-datastore:NoSchedule
Notes:
Toleration for the Cassandra Pods can be defined in the Apigee Hybrid overrides.yaml file under the cassandra property, as shown below.
cassandra:
hostNetwork: false
datacenter: "us-central1"
. . .
tolerations:
- key: "workload"
operator: "Equal"
value: "cassandra-datastore"
effect: "NoSchedule"
Using the above-mentioned configuration, it can be ensured that the apigee-data Node Pool only schedules the Cassandra workloads.
For more details on the Apigee Hybrid Runtime Components which support Node Taints and Tolerations, check out the official documentation.
Note: In Apigee Hybrid overrides.yaml file, tolerations should always be supplied as an array of objects, as shown in the configuration above i.e. the properties should be defined like cassandra.tolerations[].key, cassandra.tolerations[].operator, etc.
Conclusion
Using Node Taints and Tolerations could help ensure that the workloads are getting scheduled on the right Worker Nodes. This configuration could be used in conjunction with Node Selector and Node Labels. For Apigee Hybrid, it is not mandatory to use Node Taints and Tolerations, however, their use could help enforce some checks and keep critical workloads safe from resource starvation and scheduling issues.
Note: If any custom DaemonSets are to be run on the apigee-data Node Pool (the one which has the taint) for logging/monitoring/etc. purposes, tolerations should be specified for those DaemonSets as well. For example, if a user wants to use Apigee Logger for telemetry purposes, tolerations should be configured for Apigee Logger as well, as shown below.
logger:
enabled: true
. . .
tolerations:
- key: "workload"
operator: "Equal"
value: "cassandra-datastore"
effect: "NoSchedule"
Specifying tolerations like this should not impact the Apigee Logger Pods that are to be scheduled on the apigee-runtime Node Pool (assuming the apigee-runtime Node Pool doesn’t have any taints applied). For the DaemonSets which are required for proper functioning of Kubernetes Cluster, like the ones coming by default as part of kube-system, generally speaking, there would be no need for adding any additional tolerations because such DaemonSets have some pre-configured tolerations which let them get scheduled on all the Worker Nodes; it is highly recommended to evaluate this behaviour in some lower/Non-Production ecosystem before deploying in Production.
This article is the first part of a multi-part series. The next article will cover advanced concepts like tolerationSeconds and delve into some of the leading effective scheduling practices for Apigee Hybrid.
LinkedIn
Twitter
