All backend services are in UNHEALTHY in gke ingress

I'm getting unhealthy issue in ingress deployment. Its Autopilot Cluster in GKE, I'm using and its have 4 nodes, and All services are up and running.

7 Service have same manifest file, and also I used ingress and ssl domain.

I have deploy total 7 ingress out of 5 service its showing Status as All backend services are in UNHEALTHY.


And for 2 service which i deploy its showing Status OK.

Here is the screenshot which showing healthy.

Here is my yaml file of one service which I deploy and it showing OK status:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "unhealthy"
namespace: "xxxxx"
labels:
app: "unhealthy"
spec:
replicas: 2
selector:
matchLabels:
app: "unhealthy"
template:
metadata:
labels:
app: "unhealthy"
spec:
containers:
- name: "unhealthy"
image: "xxxxxxxxxxxxxxxxxxx"
imagePullPolicy: Always
ports:
- containerPort: 8080
# readinessProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# periodSeconds: 5
# livenessProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# periodSeconds: 5
# startupProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# failureThreshold: 0
# periodSeconds: 10
env:
- name: JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: ""healthy"_API_URL"
value: "https://xxx.dev.xxxx.ai"

---
apiVersion: v1
kind: Service
metadata:
name: "healthy"
namespace: xxxx
labels:
app: "healthy"
spec:
type: ClusterIP
selector:
app: "healthy"
ports:
- port: 8080
targetPort: 8080
protocol: TCP

Here is my yaml file of one service which i deploy and it showing unhealthy status

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "test"
namespace: "xxxxx"
labels:
app: "test"
spec:
replicas: 2
selector:
matchLabels:
app: "test"
template:
metadata:
labels:
app: "unhealthy"
spec:
containers:
- name: "unhealthy"
image: "xxxxxxxxxxxxxxxxxxx"
imagePullPolicy: Always
ports:
- containerPort: 8080
# readinessProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# periodSeconds: 5
# livenessProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# periodSeconds: 5
# startupProbe:
# httpGet:
# path: /actuator/health
# port: 8080
# initialDelaySeconds: 30
# failureThreshold: 0
# periodSeconds: 10
env:
- name: JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: ""unhealthy"_API_URL"
value: "https://xxx.dev.xxxx.ai"

---
apiVersion: v1
kind: Service
metadata:
name: "unhealthy"
namespace: xxxx
labels:
app: "unhealthy"
spec:
type: ClusterIP
selector:
app: "unhealthy"
ports:
- port: 8080
targetPort: 8080
protocol: TCP

I read this docs:

https://cloud.google.com/sdk/gcloud/reference/compute/backend-services/get-health https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#health_checks But didnt got understand where im wrong and what i need to do to make healthy from unhealthy

Please, can you tell me where I'm wrong what need to do make 5 service connect to healthy.

When i hit on browser i got 502 from service

Normal LoadBalancerNegWithoutHealthCheck 2m14s neg-readiness-reflector Pod is in NEG "Key{\"k8s1-600f13cf-default-my-svc-8080-f82bf741\", zone: \"europe-west1-b\"}". NEG is not attached to any Backend
Service with health checking. Marking condition "cloud.google.com/load-balancer-neg-ready" to True.
And
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds

If u need ssl and ingress yaml file that also i can share and im not using readinessProbe or livenessProbe or startupProbe for all this 7 service. If required in unhealthy deployment to make healthy then let me know i will add those and reapply it in K8s cluster

1 1 606
1 REPLY 1

Hi @mohdrashid ,

The readiness probe checks if your application is ready to serve traffic, liveness probe on the other hand, checks if your application is still running. Lastly, startup probe checks if your application has started successfully.

You'd have to uncomment the readiness and liveness probes your unhealthy deployment yaml file. You can check my sample/corrected yaml file:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "test"
namespace: "xxxxx"
labels:
app: "test"
spec:
replicas: 2
selector:
matchLabels:
app: "test"
template:
metadata:
labels:
app: "unhealthy"
spec:
containers:
- name: "unhealthy"
image: "xxxxxxxxxxxxxxxxxxx"
imagePullPolicy: Always
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 30
periodSeconds: 5
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 30
periodSeconds: 5
startupProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 30
failureThreshold: 0
periodSeconds: 10
env:
- name: JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: ""unhealthy"_API_URL"
value: "https://xxx.dev.xxxx.ai"