The Envoy proxy can either be deployed on a virtual machine/container in standalone mode or it can be deployed on Kubernetes using Istio Service Mesh. In standalone mode Envoy proxy configuration needs to be manually configured using a configuration file and with Istio the Envoy proxy is configured via Istio Service Mesh using Envoy Filters.
Apigee has developed an Envoy proxy Adapter which can be used for intercepting traffic sent through Envoy proxy and applying security, capturing analytics, and other API Management features that Apigee Edge provides. In this article I will explain how Apigee Envoy Adapter can be applied to an Envoy proxy deployed on Kubernetes with Istio and integrate it with Apigee Edge Cloud. Following are the high level steps that we need to follow for this process:
1. Install Google Cloud (gcloud) CLI
Install Google Cloud (gcloud) CLI by following information given on the following link if it not installed already: https://cloud.google.com/sdk/docs/install
2. Update Google Cloud (gcloud) CLI
Update Google Cloud (gcloud) CLI to its latest release:
gcloud components update
3. Install Kubernetes CLI (kubectl):
gcloud components install kubectl
1. Select an existing GCP project or create a new one, set GCP project id, region, zone and initialize Google Cloud CLI (gcloud):
export PROJECT_ID=#project id export REGION=#region export ZONE=#zone gcloud config set project $PROJECT_ID gcloud config set compute/region $REGION gcloud config set compute/zone $ZONE
2. Create a Kubernetes Cluster by enabling Istio addon:
export CLUSTER_NAME="$(whoami)-apigee-envoy-1" gcloud beta container clusters create $CLUSTER_NAME \--machine-type "e2-standard-4" --num-nodes "3" --enable-autoscaling --min-nodes "3" --max-nodes "6" --addons HorizontalPodAutoscaling,HttpLoadBalancing,Istio --istio-config auth=MTLS_PERMISSIVE
3. Get credentials and configure Kubernetes CLI:
gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT_ID
1. Set Apigee Edge Cloud environment context using following environment variables:
export ORG=#Edge cloud organization export ENV=#Edge cloud environment export USER=#Apigee username export PASSWORD=$#Apigee password
2. Provision Apigee Remote Service using below command, set MFA value if required:
export MFA=#Multi-Factor Authentication (MFA) code if MFA is enabled for above user apigee-remote-service-cli provision --legacy --username $USER --password $PASSWORD --organization $ORG --environment $ENV --mfa $MFA > config.yaml
Above command will create and deploy an API proxy with the name "remote-service" in the given Apigee Edge Cloud environment. The generated config.yaml file will contain a Kubernetes ConfigMap for deploying Apigee Envoy Adapter (apigee-remote-service-envoy) on Kubernetes.
3. Create an API product in the Apigee Edge with the name “HTTP-BIN”:
4. Create a Developer Application and connect it to the above API product. Export the API Key to an environment variable:
export API_KEY=#value
5. Generate sample Apigee Evoy Adapter Kubernetes/Istio manifests:
apigee-remote-service-cli samples create -c config.yaml
Above command will generate following files inside samples/ folder:
1. Deploy httpbin Kubernetes Deployment and Kubernetes Service:
kubectl apply -f samples/httpbin.yaml
2. Verify the status of the httpbin pod and service:
kubectl get pods
3. Verify the status of the httpbin pod and service:
kubectl get services
4. Expose httpbin service via a Load Balancer:
kubectl expose deployment httpbin --type=LoadBalancer --name=httpbin-external-service
5. Wait until the load balancer get activated:
kubectl get services/httpbin-external-service -w
6. Verify Sample Service:
Export load balancer external ip address to an environment variable and send an API request to verify httbin service deployed on the Kubernetes cluster:
export EXTERNAL_IP=$(kubectl get services/httpbin-external-service -o jsonpath="{.status.loadBalancer.ingress[*].ip}") curl -i --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get
An example output:
HTTP/1.1 200 OK server: istio-envoy date: Mon, 21 Sep 2020 07:10:04 GMT content-type: application/json content-length: 337 access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 2 x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/* { "args": {}, "headers": { "Accept": "*/*", "Content-Length": "0", "Host": "***", "User-Agent": "curl/7.64.1", "X-B3-Sampled": "1", "X-B3-Spanid": "***", "X-B3-Traceid": "***" }, "origin": "127.0.0.1", "url": "http://***/get" }
Note that httpbin service should now be accessible through the load balancer. The message flow of the above API request is as follows:
Client -> GCP Load Balancer -> httpbin Service -> httpbin Pod
1. Bind Kubernetes/Istio httpbin service to the above created API product. Binding associates a service deployed to the Istio mesh with an Apigee API product (Reference😞
apigee-remote-service-cli bindings add httpbin HTTP-BIN -o $ORG -e $ENV -u $USER -p $PASSWORD --mfa $MFA -c config.yaml
2. Deploy Apigee Envoy Adapter (apigee-remote-service-envoy) Kubernetes Deployment and Service:
kubectl create namespace apigee kubectl apply -f config.yaml kubectl apply -f samples/apigee-envoy-adapter.yaml
3. Verify the status of the Apigee Envoy Adapter deployment:
kubectl -n=apigee get deployments
4. Apply the Envoy Filter generated for engaging Apigee Envoy Adapter for Kubernetes services:
kubectl apply -f samples/envoyfilter-sidecar.yaml
5. Verify Envoy Filter creation:
kubectl get envoyfilters/apigee-remote-httpbin
6. Enable Istio Sidecar Injection for the default (or any other preferred) namespace in the Kubernetes cluster:
NAMESPACE=default kubectl label namespace $NAMESPACE istio-injection=enabled
Once this feature is enabled, Istio will inject Envoy proxy container to all pods created in the default namespace from this point onwards for intercepting incoming and outgoing traffic. Any existing pods will not get affected:
kubectl get pods -l app=httpbin NAME READY STATUS RESTARTS AGE httpbin-56f8d695dc-5sv2z 1/1 Running 0 5m
7. Delete httpbin pod and let it get re-created by the httpbin deployment to inject the Envoy proxy container:
kubectl delete pod -l app=httpbin
8. Verify the httpbin pod status:
kubectl get pods -l app=httpbin
Now, the READY column of the httpbin pod should show "2/2" indicating the injection of the Envoy proxy container.
kubectl get pods -l app=httpbin NAME READY STATUS RESTARTS AGE httpbin-56f8d695dc-5sv2z 2/2 Running 0 1m
In addition, the following command should list both httpbin container image and Envoy proxy container image:
kubectl get pods -l app=httpbin -o=json | jq '.items[].spec.containers[].image'
An example output:
"docker.io/kennethreitz/httpbin" "gke.gcr.io/istio/proxyv2:1.4.10-gke.5"
9. Tail Apigee Envoy Adapter (apigee-remote-service-envoy) logs on a separate terminal. On these logs, we could see API request verification information.
kubectl -n=apigee logs -f -l app=apigee-remote-service-envoy
10. Trace Apigee Remote Service API Proxy:
Log into Apigee Edge UI and start a Trace session in remote-service API proxy. In this trace session we could see API requests sent by Apigee Envoy Adapter for listing API products and verifying API keys.
11. Verify Apigee Envoy Adapter:
11.1 Try to access the httpbin service via the Load Balancer. It should return a HTTP 403 Forbidden error because of applying the above Envoy Filter and not specifying an API key in the API request:
curl -i --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get HTTP/1.1 403 Forbidden date: Fri, 25 Sep 2020 06:04:54 GMT server: istio-envoy x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/* content-length: 0
11.2 Now, use the API key generated by the Developer Application and make another API request:
curl -i -H "x-api-key: $API_KEY" --resolve httpbin:80:$EXTERNAL_IP http://httpbin/get
This should now return a successful response:
HTTP/1.1 200 OK server: istio-envoy date: Fri, 25 Sep 2020 06:36:25 GMT content-type: application/json content-length: 784 access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 5 x-envoy-decorator-operation: httpbin.default.svc.cluster.local:80/* { "args": {}, "headers": { "Accept": "*/*", "Content-Length": "0", "Host": "httpbin", "User-Agent": "curl/7.64.1", "X-Api-Key": "xxx", "X-Apigee-Accesstoken": "", "X-Apigee-Api": "httpbin", "X-Apigee-Apiproducts": "HTTP Bin", "X-Apigee-Application": "HTTP Bin", "X-Apigee-Authorized": "true", "X-Apigee-Clientid": "***", "X-Apigee-Developeremail": "***", "X-Apigee-Environment": "test", "X-Apigee-Organization": "***", "X-Apigee-Scope": "", "X-B3-Sampled": "0", "X-B3-Spanid": "***", "X-B3-Traceid": "***" }, "origin": "127.0.0.1", "url": "http://httpbin/get" }
At this point if you are still seeing the HTTP 403 Forbidden error, check apigee-remote-service-envoy logs to identify the reason:
kubectl -n=apigee logs -l app=apigee-remote-service-envoy
Hi Imesh,
Very nice article and very useful.
you mentioned the export as below
export API_KEY=#value
where as in the command it is mentioned as
"x-api-key: $APIKEY"
Many thanks Phani for your feedback and pointing out the typo in the API_KEY environment variable. I just fixed it by renaming APIKEY to API_KEY in the curl command HTTP header. Thanks again!
Hi @Imesh Gunaratne
Thanks for the article. I'm able to set it up and run it. Sidecar envoy throws 403 without api key header and 200 after adding api key header.
However I am not able to pass the api calls via JWT. I am getting 403 Forbidden.
The curl command to API (adding JWT token as Bearer token in Authorization header):
curl -i --resolve httpbin:80:$EXTERNAL_IP http://httpbin/headers -H "Authorization:Bearer $TOKEN"
I am getting token through cli
./apigee-remote-service-cli token create -c config.yaml --id $KEY --secret $SECRET -o $ORG -e $ENV
./apigee-remote-service-cli -c config.yaml token inspect <<< $TOKEN
The error log from remote-service envoy is as below:
2021-01-11T12:22:48.213ZDEBUGauth/auth.go:98Authenticate: key: , claims: map[string]interface {}(nil)<br>2021-01-11T12:22:48.213ZDEBUGauth/auth.go:159Authenticate error: &auth.Context{Context:(*server.Handler)(0xc00019a6c0), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [missing authentication]<br>2021-01-11T12:22:48.213ZDEBUGserver/authorization.go:205sending denied: UNAUTHENTICATED<br>2021-01-11T12:22:48.762ZDEBUGserver/header_context.go:68No context header x-apigee-api, using target header: :authority
Do I need to explicitly enable JWT as Authorization header in envoy proxy?
Hi @vivek yadav
Thanks for your feedback! We need to configure Envoy JWT Authentication Filter for validating JWT tokens. Once JWT token is authenticated, the Envoy ext-authz filter sends the request headers and JWT to apigee-remote-service-envoy. It matches the JWT's api_product_list and scope claims against Apigee API Products to authorize it against the target of the request.
According to the error "missing authentication" I think Envoy JWT Authentication Filter may have not configured. Thanks!
Reference: https://docs.apigee.com/api-platform/envoy-adapter/v1.4.x/operation#overview
Hi @imesh , have implemented all the steps given in article however sidecar envoy is not throwing 403 it always respond back with 200. Any suggestions?
@devNikhil We have written a playbook to troubleshoot HTTP 403 errors raised by the Envoy Proxy when integrated with Apigee Envoy Adapter:
Can you please try go through it and let us know your observations? Thanks!
hi @imesh
kubectl apply -f samples/envoyfilter-sidecar.yaml
i am getting the following. How do i resolve it?
error: unable to recognize "samples/envoyfilter-sidecar.yaml": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
Hi @imesh will this tutorial works for Apigee X too?
Thanks,
Answering myself, the preferred and updated way is to follow this other tutorial:
https://cloud.google.com/apigee/docs/api-platform/envoy-adapter/v2.0.x/example-hybrid
The only part you should do extra is create the "apigee" namespace in your own GKE, before use it in the Apigee X.
@imesh Thank you for the post. Do you know if the Apigee Envoy Adapter can be configured with other service meshes outside of Istio that use Envoy as the sidecar proxy, such as Consul?