- I am installing on GKE < 1.18.17-gke.100 > on GCP
- apigee version 1.5
- cert-manager version :
- Init script gets completed successfully, if no network-policies are applied between cert-manager and apigee-system namespace.
- But when applying networkpolicies , it start failing with below error :
hybrid-files~git$ $APIGEECTL_HOME/apigeectl init -f overrides/overrides.yaml
Parsing file: config/values.yaml
Parsing file: overrides/overrides.yaml
Invoking "kubectl apply" with pre-init YAML config...
priorityclass.scheduling.k8s.io/high-priority configured
namespace/apigee-system unchanged
namespace/apigee unchanged
serviceaccount/apigee-init unchanged
clusterrole.rbac.authorization.k8s.io/apigee-init unchanged
clusterrolebinding.rbac.authorization.k8s.io/apigee-init unchanged
peerauthentication.security.istio.io/apigee-system unchanged
peerauthentication.security.istio.io/apigee unchanged
secret/apigee-datastore-default-creds unchanged
secret/apigee-redis-default-creds unchanged
Waiting for cert manager's pods to be running before proceeding (usually its webhook takes ~3 mins to come up completely)...
Please re-run *init* again if this times out....(Also, check the health of cert manager components before the re-run)...
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
job.batch "apigee-resources-install" deleted
All init jobs are complete, proceeding...
Invoking "kubectl apply" with final init YAML config...
envoyfilter.networking.istio.io/apigee-envoyfilter unchanged
customresourcedefinition.apiextensions.k8s.io/apigeedatastores.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeedeployments.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeeenvironments.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeeorganizations.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeeredis.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeerouteconfigs.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeeroutes.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/apigeetelemetries.apigee.cloud.google.com configured
customresourcedefinition.apiextensions.k8s.io/cassandradatareplications.apigee.cloud.google.com configured
role.rbac.authorization.k8s.io/apigee-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/apigee-manager-role configured
clusterrole.rbac.authorization.k8s.io/apigee-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/apigee-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/apigee-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/apigee-proxy-rolebinding unchanged
service/apigee-controller-manager-metrics-service unchanged
service/apigee-webhook-service unchanged
deployment.apps/apigee-controller-manager configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/apigee-mutating-webhook-configuration configured
validatingwebhookconfiguration.admissionregistration.k8s.io/apigee-validating-webhook-configuration configured
serviceaccount/apigee unchanged
clusterrole.rbac.authorization.k8s.io/apigee unchanged
clusterrolebinding.rbac.authorization.k8s.io/apigee unchanged
configmap/apigee-config configured
job.batch/apigee-resources-install created
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: dial tcp 10.162.81.45:10250: i/o timeout
Error: unable to exec kubectl: error applying k8s config via kubectl: exit status 1
--
Netpol in apigee-system to talk to cert-manager is :
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
creationTimestamp: "2021-07-24T12:18:37Z"
generation: 1
name: allow-local-traffic-cert-manager
namespace: apigee-system
resourceVersion: "43325458"
selfLink: /apis/networking.k8s.io/v1/namespaces/apigee-system/networkpolicies/allow-local-traffic-cert-manager
uid: 5ce829e9-8844-4e43-bc61-04abd43e3b2b
spec:
egress:
- to:
- namespaceSelector:
matchLabels:
app_name: cert-manager
ingress:
- from:
- namespaceSelector:
matchLabels:
app_name: cert-manager
podSelector: {}
policyTypes:
- Ingress
- Egress
===
Netpol in cert-manager namespace to communicate to apigee-system
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
creationTimestamp: "2021-07-25T15:16:47Z"
generation: 1
name: allow-local-traffic-default-apigee-system-latest
namespace: cert-manager
resourceVersion: "44308090"
selfLink: /apis/networking.k8s.io/v1/namespaces/cert-manager/networkpolicies/allow-local-traffic-default-apigee-system-latest
uid: 62a523c8-ecdc-4195-a87f-0eb49a7f6407
spec:
egress:
- to:
- namespaceSelector:
matchLabels:
app_name: apigee-system
ingress:
- from:
- namespaceSelector:
matchLabels:
app_name: apigee-system
podSelector: {}
policyTypes:
- Ingress
- Egress
--> Note: Both namespace have required label added correctly.
Below marked testing is done to cross-check the netpol configuration between apigee-system and cert-manager & test passed successfully.
At same time, i have deployed a sample application apigee-system namespace to check the connectivity to cert-manager "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s"
k exec -it testconnect-546fbc99f5-fq9tc bash
+ kubectl exec -it testconnect-546fbc99f5-fq9tc bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.0#
bash-5.0#
bash-5.0#
bash-5.0#
bash-5.0#
bash-5.0# curl -kv https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s
* Trying 10.162.235.11:443...
* TCP_NODELAY set
* Connected to cert-manager-webhook.cert-manager.svc (10.162.235.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: [NONE]
* start date: Jul 24 12:03:33 2021 GMT
* expire date: Jul 31 12:03:33 2021 GMT
* issuer: CN=cert-manager-webhook-ca
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /mutate?timeout=10s HTTP/1.1
> Host: cert-manager-webhook.cert-manager.svc
> User-Agent: curl/7.67.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Sun, 25 Jul 2021 15:44:26 GMT
< Content-Length: 0
<
* Connection #0 to host cert-manager-webhook.cert-manager.svc left intact
bash-5.0#
bash-5.0# nslookup ^C
bash-5.0# ping ^C
bash-5.0# curl -k v 10.162.81.45:10250
curl: (6) Could not resolve host: v
Client sent an HTTP request to an HTTPS server.
bash-5.0# curl -kv 10.162.81.45:10250
* Trying 10.162.81.45:10250...
* TCP_NODELAY set
* Connected to 10.162.81.45 (10.162.81.45) port 10250 (#0)
> GET / HTTP/1.1
> Host: 10.162.81.45:10250
> User-Agent: curl/7.67.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
<
Client sent an HTTP request to an HTTPS server.
* Closing connection 0
bash-5.0#
LinkedIn
Twitter
