Taints for Apigee hybrid 1.9.x

[aramkrishna6]

Hi,

For apigee hybrid 1.9.x do nodes requires taints ? document did not indicate them, let me know if missing anything.

We can remove taints and tolerations for apigee nodes (in a cluster) installation either for dev or higher environments ?

Listed link does not  indicates about same https://cloud.google.com/apigee/docs/hybrid/v1.9/configure-dedicated-nodes 

Do see taints for nodes attached for apigee nodes from cluster, hence when installing apigee get listed error.

1)kubectl get nodes

ip-xx.yy.dd.mm.ec2. Ready <none> 2d2h v1.23.x-eks-xyz
ip-xx.yy.dd.mm1.ec2.Ready <none> 2d2h v1.23.x-eks-xyz
ip-xx.yy.dd.mm3.ec2. Ready <none> 2d2h v1.23.x-eks-xyz

kubectl describe nodes <node name> -0 yaml shows taints attached to it 

2) kubectl get pod -n apigee

apigee-ingressgateway-manager-xyz 0/3 Pending 0 20h
apigee-ingressgateway-manager-xyz1 0/3 Pending 0 20h

3)   describing each pod resulted in following

cluster-autoscaler pod didn't trigger scale-up: 1 node(s) had taint {xyz.com/usage: my-services}, that the pod didn't tolerate, 4 node(s) didn't match Pod's node affinity/selector
Warning FailedScheduling  default-scheduler 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had taint {xyz.com/usage: my-services}, that the pod didn't tolerate.
Normal NotTriggerScaleUp  cluster-autoscaler pod didn't trigger scale-up: 4 node(s) didn't match Pod's node affinity/selector, 1 node(s) had taint {xyz.com/usage: my-services}, that the pod didn't tolerate

@dino @dhtx  @strebel 

[dhtx]

The affinity settings ensure pods are scheduled to the correct node pool. It's recommended not to remove these settings.

[aramkrishna6]

Agree with affinity settings but does same applies to  taints and tolerations recommendations for apigee hybrid node cluster ?

[dhtx]

It's not recommended to remove the taints and tolerations settings.

[aramkrishna6]

@dhtx  Listed doc gives importance of scheduling  using listed information.

1) The nodeSelector config section has a property called requiredForScheduling:  If set to false (the default), underlying pods will be scheduled whether or not node pools are defined with the required names.  

This list best practice as 

Note: The best practice is to set this value to requiredForScheduling:true for a production environment.

2) Docs referenced for creation of cluster (either GKE cluster or EKS cluster, which references to Amazon link) covers required fundamental  cluster requirements and  configuration, 

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ 

Recommendation for Apigee hybrid with taint-and-toleration with cluster with respect to apigee hybrid cluster was not clear.