This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

Apigee Hybrid - Enable L7 Load Balancer

‎03-21-2023 04:29 AM - edited ‎04-23-2023 06:43 PM

ganadurai
Staff
2 0 617

This article is an extension  of an existing article that details the ways Hybrid Runtime Ingress can be exposed for consumption. This article focuses on the steps that need to be executed for exposing Apigee Hybrid Ingress on HTTPS and HTTP via L7 external load balancer. 

This execution is validated on Apigee Hybrid 1.9 deployed in GKE. With minor tweaks the same can be done for other Kubernetes platforms.

Prerequisites

Complete Apigee Hybrid Installation as detailed here. Deploy an apigee proxy with base path apigee-hybrid-helloworld via management console 

Setup the following variables:

export PROJECT_ID=<Apigee Hybr id GCP Project>
export DNS_PROJECT_ID=$PROJECT_ID
export ORG_NAME=$PROJECT_ID
export ENVIRONMENT_NAME=<Environment name>
export ENV_GROUP=<Environment-group containing the environment>
export ENV_GROUP_HOSTNAME=<Hostname that will point to the XLB>
export INGRESS_NAME=<The name provided for ingressGateways.name in overrides.yaml>

# Directory locations variables used in the hybrid install as defined here
# Confirm the values are set correctly
echo $APIGEECTL_BASE
echo $APIGEECTL_HOME
echo $HYBRID_FILES
gcloud config set project $PROJECT_ID


Service definition to enable services exposed on the GKE nodes.

cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: Service

metadata:

  name: $ENV_GROUP-ingrs-service

  namespace: apigee

  annotations:

    cloud.google.com/app-protocols: '{"https":"HTTPS","http2":"HTTP"}'

    cloud.google.com/backend-config: '{"ports": {"443": "http-hc-config","80": "http-hc-config"}}'

    cloud.google.com/neg: '{"ingress": true}'

spec:

  ports:

  - name: status-port

    port: 15021

    protocol: TCP

    targetPort: 15021

  - name: http2

    port: 80

    protocol: TCP

    targetPort: 8080

  - name: https

    port: 443

    protocol: TCP

    targetPort: 8443

  selector:

    app: apigee-ingressgateway #required

    ingress_name: $INGRESS_NAME

    org: $ORG_NAME

  type: NodePort

EOF


Create static external IP address which will be assigned to the Load balancer.

gcloud compute addresses create apigee-global --global
export EXTERNAL_IP_ADDR=$(gcloud compute addresses describe apigee-global \

  --project $PROJECT_ID --format="get(address)" --global);
echo $EXTERNAL_IP_ADDR;

Create Backendconfig object, this creates the needed Healthcheck for the Load balancer backend services.

cat <<EOF | kubectl apply -f -
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: http-hc-config
  namespace: apigee
spec:
  healthCheck:
    checkIntervalSec: 15
    requestPath: /healthz/ingress
EOF


HTTP External Load balancer to HTTP Container ports

Starting with Apigee Hybrid release 1.9.1 HTTP (Port 80) Ingress is disabled. From security standpoint use of non-secure ingress should be avoided. As per release notes. "Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway."
 
The Ingress resource will create the external load balancer with the external address created in the above step.

cat <<EOF | kubectl apply -f -

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  annotations:

    kubernetes.io/ingress.global-static-ip-name: apigee-global

  name: xlb-http-apigee-ingress

  namespace: apigee

spec:

  defaultBackend:

    service:

      name: $ENV_GROUP-ingrs-service

      port:

        number: 80

EOF


ApigeeRoute definition will enable non-SNI client to reach the Ingress. The loadbalancer healthchecks does not have SNI in the request. This enables Ingress to respond to the healthcheck calls. 

cat <<EOF | kubectl apply -f -

apiVersion: apigee.cloud.google.com/v1alpha1

kind: ApigeeRoute

metadata:

  name: wildcard-gateway-apigee

  namespace: apigee

spec:

  hostnames:

  - "*"

  ports:

  - number: 80

    protocol: HTTP

  selector:

    app: apigee-ingressgateway

  enableNonSniClient: true

EOF

In the $HYBRID_FILES/overrides/overrides.yaml add the following:
additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name and run apigeectl apply.

cd $HYBRID_FILES
${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml

 

Test the endpoint:
(make sure $ENV_GROUP_HOSTNAME value is added as hostname for the Apigee environment group)

curl http://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld --resolve $ENV_GROUP_HOSTNAME:80:$EXTERNAL_IP_ADDR

 
A successful response from the above request validates the traffic is reachable from external  loadbalancer to the deployed runtime pods and executed.

HTTPS - L7 - External Loadbalancer

Cert Provisioning from Let’s Encrypt

For the purpose of this article we will use Cert-Manager issuer for Let’s Encrypt and automatically provision certificates on your Apigee hybrid ingress load balancer. Thanks to Daniel Strebel on this article to configure Let’s Encrypt cert provisioning. Some of the steps are modified from the original article to work for version 1.9

Follow this doc to setup a DNS zone. Create record set type A for the hostname (ENV_GROUP_HOSTNAME) that points to your Apigee hybrid ingress IP address (GCP address created in the prerequisites step). Create record set type CAA  for the hostname(ENV_GROUP_HOSTNAME) with Routing data value <0 issue "letsencrypt.org"> (just copy the value within the angle brackets).

gcloud config set project $PROJECT_ID
gcloud services enable --project=${PROJECT_ID} dns.googleapis.com

 

gcloud iam service-accounts create dns01-solver --display-name "dns01-solver" --project $PROJECT_ID

gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \
--member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/dns.admin

gcloud iam service-accounts keys create dns-sa-key.json \
--iam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com

 

kubectl create secret generic clouddns-dns01-solver-svc-acct \
--from-file=dns-sa-key.json -n apigee

 

cat <<EOF | kubectl apply -f -

apiVersion: cert-manager.io/v1

kind: Issuer

metadata:

  name: cloud-dns-issuer

  namespace: apigee

spec:

  acme:

    email: $YOUR_EMAIL_ADDRESS

    server: https://acme-v02.api.letsencrypt.org/directory

    privateKeySecretRef:

      name: cloud-dns-issuer-account-key

    solvers:

    - dns01:

        cloudDNS:

          project: $DNS_PROJECT_ID

          serviceAccountSecretRef:

            name: clouddns-dns01-solver-svc-acct

            key: dns-sa-key.json

EOF

 

cat <<EOF | kubectl apply -f -

apiVersion: cert-manager.io/v1

kind: Certificate

metadata:

  name: cert-manager-$PROJECT_ID-default

  namespace: apigee

spec:

  secretName: cert-manager-$PROJECT_ID-tls

  issuerRef:

    name: cloud-dns-issuer

  commonName: '*.$ENV_GROUP_HOSTNAME'

  dnsNames:

  - $ENV_GROUP_HOSTNAME

  - '*.$ENV_GROUP_HOSTNAME'

EOF

 

(wait for 2 mts, the below command should show status as True)

 

kubectl -n apigee get Certificate cert-manager-$PROJECT_ID-default
kubectl get secret cert-manager-$PROJECT_ID-tls -n apigee -o yaml

 
If the Certificate Kubernetes object does not have the “Ready” attribute marked as “True” beyond 5 minutes, use the below links as source to troubleshoot the issue:

https://cert-manager.io/docs/troubleshooting/acme/
https://cert-manager.io/v1.6-docs/faq/acme/
https://www.techrepublic.com/article/how-to-add-a-certificate-authority-authorization-record-in-goog...

 
HTTPS External Load Balancer to HTTP Container ports

The Ingress resource will create the external load balancer with the external address created in the above step. (Confirm the service referenced in this Ingress is created as defined in the steps above)

cat <<EOF | kubectl apply -f -

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  annotations:

    kubernetes.io/ingress.allow-http: "false"

    kubernetes.io/ingress.global-static-ip-name: apigee-global

  name: xlb-https-apigee-ingress

  namespace: apigee

spec:

  defaultBackend:

    service:

      name: $ENV_GROUP-ingrs-service

      port:

        number: 80

  tls:

  - hosts:

      - $ENV_GROUP_HOSTNAME

    secretName: cert-manager-$PROJECT_ID-tls

EOF

 

Edit ApigeeRoute to indicate the container is enabled to accept traffic on port and to enable non-SNI clients (healthchecks from loadbalancer) 

cat <<EOF | kubectl apply -f -

apiVersion: apigee.cloud.google.com/v1alpha1

kind: ApigeeRoute

metadata:

  name: wildcard-gateway-apigee

  namespace: apigee

spec:

  hostnames:

  - "*"

  ports:

  - number: 80

    protocol: HTTP

  selector:

    app: apigee-ingressgateway

  enableNonSniClient: true

EOF

 
In the overrides.yaml add additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name.

cd $HYBRID_FILES
${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml


Test the endpoint

curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld


A successful response from the above request validates the traffic is reachable from external  loadbalancer to the deployed runtime pods and executed.

 

HTTPS External Load Balancer to HTTPS Container ports

If you are using the cert generated by “LetsEncrypt” as detailed in this doc. Use the below to extract the cert and key and initialize the variables.

cd $HYBRID_FILES/certs

kubectl -n apigee get secret cert-manager-$PROJECT_ID-tls -o json | \
jq -r '.data."tls.key"' | \
base64 -d > $HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.key

kubectl -n apigee get secret cert-manager-$PROJECT_ID-tls -o json | \
jq -r '.data."tls.crt"' | \
base64 -d > $HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.pem

export CERT_FILE=$HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.pem
export KEY_FILE=$HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.key

Update the overrides file for the env virtualhost section, the cert and the key file. Replace the sslCertPath and sslKeyPath values as below and run apigeectl apply on the overrides file. 

virtualhosts:

- name: ENVIRONMENT_GROUP_NAME

  selector:

    app: apigee-ingressgateway

    ingress_name: INGRESS_NAME

  sslCertPath: $CERT_FILE

  sslKeyPath: $KEY_FILE

 

cd $HYBRID_FILES
${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml

 

cat <<EOF | kubectl apply -f -

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  annotations:

    kubernetes.io/ingress.allow-http: "false"

    kubernetes.io/ingress.global-static-ip-name: apigee-global

  name: xlb-https-apigee-ingress

  namespace: apigee

spec:

  defaultBackend:

    service:

      name: $ENV_GROUP-ingrs-service

      port:

        number: 443

  tls:

  - hosts:

      - $ENV_GROUP_HOSTNAME

    secretName: cert-manager-$PROJECT_ID-tls

EOF


Edit ApigeeRoute “wildcard-gateway-apigee” to contain 443 in the port list. This makes the container to accept traffic from loadbalancer only on port 443.

cat <<EOF | kubectl apply -f -

apiVersion: apigee.cloud.google.com/v1alpha1

kind: ApigeeRoute

metadata:

  name: wildcard-gateway-apigee

  namespace: apigee

spec:

  hostnames:

  - "*"

  ports:

  - number: 443

    protocol: HTTPS

    tls:
      credentialName: $PROJECT_ID-$ENV_GROUP

      mode: SIMPLE

  selector:

    app: apigee-ingressgateway

  enableNonSniClient: true

EOF

 

In the overrides.yaml add additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name.

cd $HYBRID_FILES
${APIGEECTL_HOME}/apigeectl apply -f overrides/overrides.yaml


Test the endpoint

curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld

 

A successful response from the above request validates the traffic is reachable from external  loadbalancer to the deployed runtime pods and executed.

Topic Labels
Contributors
Version history
Last update:
‎04-23-2023 06:43 PM
Updated by: