The Envoy proxy can either be deployed on a virtual machine/container in standalone mode or it can be deployed on Kubernetes using Istio Service Mesh. In standalone mode Envoy proxy configuration needs to be manually configured using a configuration file and with Istio the Envoy proxy is configured via Istio Service Mesh using Envoy Filters.
Apigee has developed an Envoy proxy Adapter which can be used for intercepting traffic sent through Envoy proxy and applying security, capturing analytics, and other API Management features that Apigee Edge provides. In this article I will explain how Apigee Envoy Adapter can be applied to an Envoy proxy deployed on Kubernetes with Istio and integrate it with Apigee Edge Cloud. Following are the high level steps that we need to follow for this process:
- Install Prerequisites
- Create a Kubernetes Cluster
- Inititalize Apigee Envoy Adapter
- Deploy The Sample Service
- Deploy Apigee Envoy Adapter
1. Install Prerequisites
1. Install Google Cloud (gcloud) CLI
Install Google Cloud (gcloud) CLI by following information given on the following link if it not installed already: https://cloud.google.com/sdk/docs/install
2. Update Google Cloud (gcloud) CLI
Update Google Cloud (gcloud) CLI to its latest release:
gcloud components update
3. Install Kubernetes CLI (kubectl):
gcloud components install kubectl
2. Create a Kubernetes Cluster
1. Select an existing GCP project or create a new one, set GCP project id, region, zone and initialize Google Cloud CLI (gcloud):
export PROJECT_ID=#project id
export REGION=#region
export ZONE=#zone
gcloud config set project $PROJECT_ID
gcloud config set compute/region $REGION
gcloud config set compute/zone $ZONE
2. Create a Kubernetes Cluster by enabling Istio addon:
export CLUSTER_NAME="$(whoami)-apigee-envoy-1"
gcloud beta container clusters create $CLUSTER_NAME \--machine-type "e2-standard-4" --num-nodes "3" --enable-autoscaling --min-nodes "3" --max-nodes "6" --addons HorizontalPodAutoscaling,HttpLoadBalancing,Istio --istio-config auth=MTLS_PERMISSIVE
3. Get credentials and configure Kubernetes CLI:
gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT_ID
3. Inititalize Apigee Envoy Adapter
1. Set Apigee Edge Cloud environment context using following environment variables:
export ORG=#Edge cloud organization
export ENV=#Edge cloud environment
export USER=#Apigee username
export PASSWORD=$#Apigee password
2. Provision Apigee Remote Service using below command, set MFA value if required:
export MFA=#Multi-Factor Authentication (MFA) code if MFA is enabled for above user
apigee-remote-service-cli provision --legacy --username $USER --password $PASSWORD --organization $ORG --environment $ENV --mfa $MFA > config.yaml
Above command will create and deploy an API proxy with the name "remote-service" in the given Apigee Edge Cloud environment. The generated config.yaml file will contain a Kubernetes ConfigMap for deploying Apigee Envoy Adapter (apigee-remote-service-envoy) on Kubernetes.
3. Create an API product in the Apigee Edge with the name “HTTP-BIN”:
- Select the relevant Edge Cloud environment
- Add “remote-service” API proxy and "/verifyApiKey" "/get", "/headers" paths to it. The path "/verifyApiKey" is required to be granted for allowing Apigee Envoy Adapter for verifying API keys. Other paths "/get" and "/headers" are API proxy/target endpoint specific.
- Add “httpbin” as a target server hostname under the “Apigee Remote Service Targets” section of the API product.
- Note that Apigee Envoy Adapter will only allow traffic sent to hostnames added in the “Apigee Remote Service Targets” section. In this tutorial we will be deploying a service with the hostname "httpbin" on the Kubernetes cluster and Apigee Envoy Adapter will be used for intercepting that traffic.
4. Create a Developer Application and connect it to the above API product. Export the API Key to an environment variable:
export API_KEY=#value
5. Generate sample Apigee Evoy Adapter Kubernetes/Istio manifests:
apigee-remote-service-cli samples create -c config.yaml
Above command will generate following files inside samples/ folder:
- apigee-envoy-adapter.yaml
- A kubernetes deployment and a service for deploying Apigee Envoy Adapter
- httpbin.yaml
- A kubernetes deployment and a service for deploying an example httpbin service
- envoyfilter-sidecar.yaml
- An Envoy filter for applying Apigee Envoy Adapter as a sidecar for services deployed on Kubernetes
- request-authentication.yaml
- An Istio RequestAuthentication definition for applying JWT authentication
4. Deploy The Sample Service
1. Deploy httpbin Kubernetes Deployment and Kubernetes Service:
kubectl apply -f samples/httpbin.yaml
2. Verify the status of the httpbin pod and service:
kubectl get pods
3. Verify the status of the httpbin pod and service:
kubectl get services
4. Expose httpbin service via a Load Balancer:
kubectl expose deployment httpbin --type=LoadBalancer --name=httpbin-external-service
5. Wait until the load balancer get activated:
kubectl get services/httpbin-external-service -w
6. Verify Sample Service:
Export load balancer external ip address to an environment variable and send an API request to verify httbin service deployed on the Kubernetes cluster:
export EXTERNAL_IP=$(kubectl get services/httpbin-external-service -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
curl -i --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get
An example output:
HTTP/1.1 200 OK
server: istio-envoy
date: Mon, 21 Sep 2020 07:10:04 GMT
content-type: application/json
content-length: 337
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 2
x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/*
{
"args": {},
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "***",
"User-Agent": "curl/7.64.1",
"X-B3-Sampled": "1",
"X-B3-Spanid": "***",
"X-B3-Traceid": "***"
},
"origin": "127.0.0.1",
"url": "http://***/get"
}
Note that httpbin service should now be accessible through the load balancer. The message flow of the above API request is as follows:
Client -> GCP Load Balancer -> httpbin Service -> httpbin Pod
5. Deploy Apigee Envoy Adapter
1. Bind Kubernetes/Istio httpbin service to the above created API product. Binding associates a service deployed to the Istio mesh with an Apigee API product (Reference😞
apigee-remote-service-cli bindings add httpbin HTTP-BIN -o $ORG -e $ENV -u $USER -p $PASSWORD --mfa $MFA -c config.yaml
2. Deploy Apigee Envoy Adapter (apigee-remote-service-envoy) Kubernetes Deployment and Service:
kubectl create namespace apigee
kubectl apply -f config.yaml
kubectl apply -f samples/apigee-envoy-adapter.yaml
3. Verify the status of the Apigee Envoy Adapter deployment:
kubectl -n=apigee get deployments
4. Apply the Envoy Filter generated for engaging Apigee Envoy Adapter for Kubernetes services:
kubectl apply -f samples/envoyfilter-sidecar.yaml
5. Verify Envoy Filter creation:
kubectl get envoyfilters/apigee-remote-httpbin
6. Enable Istio Sidecar Injection for the default (or any other preferred) namespace in the Kubernetes cluster:
NAMESPACE=default
kubectl label namespace $NAMESPACE istio-injection=enabled
Once this feature is enabled, Istio will inject Envoy proxy container to all pods created in the default namespace from this point onwards for intercepting incoming and outgoing traffic. Any existing pods will not get affected:
kubectl get pods -l app=httpbin
NAME READY STATUS RESTARTS AGE
httpbin-56f8d695dc-5sv2z 1/1 Running 0 5m
7. Delete httpbin pod and let it get re-created by the httpbin deployment to inject the Envoy proxy container:
kubectl delete pod -l app=httpbin
8. Verify the httpbin pod status:
kubectl get pods -l app=httpbin
Now, the READY column of the httpbin pod should show "2/2" indicating the injection of the Envoy proxy container.
kubectl get pods -l app=httpbin
NAME READY STATUS RESTARTS AGE
httpbin-56f8d695dc-5sv2z 2/2 Running 0 1m
In addition, the following command should list both httpbin container image and Envoy proxy container image:
kubectl get pods -l app=httpbin -o=json | jq '.items[].spec.containers[].image'
An example output:
"docker.io/kennethreitz/httpbin"
"gke.gcr.io/istio/proxyv2:1.4.10-gke.5"
9. Tail Apigee Envoy Adapter (apigee-remote-service-envoy) logs on a separate terminal. On these logs, we could see API request verification information.
kubectl -n=apigee logs -f -l app=apigee-remote-service-envoy
10. Trace Apigee Remote Service API Proxy:
Log into Apigee Edge UI and start a Trace session in remote-service API proxy. In this trace session we could see API requests sent by Apigee Envoy Adapter for listing API products and verifying API keys.
11. Verify Apigee Envoy Adapter:
11.1 Try to access the httpbin service via the Load Balancer. It should return a HTTP 403 Forbidden error because of applying the above Envoy Filter and not specifying an API key in the API request:
curl -i --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get
HTTP/1.1 403 Forbidden
date: Fri, 25 Sep 2020 06:04:54 GMT
server: istio-envoy
x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/*
content-length: 0
11.2 Now, use the API key generated by the Developer Application and make another API request:
curl -i -H "x-api-key: $API_KEY" --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get
This should now return a successful response:
HTTP/1.1 200 OK
server: istio-envoy
date: Fri, 25 Sep 2020 06:36:25 GMT
content-type: application/json
content-length: 784
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 5
x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/*
{
"args": {},
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "httpbin",
"User-Agent": "curl/7.64.1",
"X-Api-Key": "xxx",
"X-Apigee-Accesstoken": "",
"X-Apigee-Api": "httpbin",
"X-Apigee-Apiproducts": "HTTP Bin",
"X-Apigee-Application": "HTTP Bin",
"X-Apigee-Authorized": "true",
"X-Apigee-Clientid": "***",
"X-Apigee-Developeremail": "***",
"X-Apigee-Environment": "test",
"X-Apigee-Organization": "***",
"X-Apigee-Scope": "",
"X-B3-Sampled": "0",
"X-B3-Spanid": "***",
"X-B3-Traceid": "***"
},
"origin": "127.0.0.1",
"url": "http://httpbin/get"
}
At this point if you are still seeing the HTTP 403 Forbidden error, check apigee-remote-service-envoy logs to identify the reason:
kubectl -n=apigee logs -l app=apigee-remote-service-envoy
LinkedIn
Twitter
