How to Troubleshoot HTTP 403 Forbidden Error in Apigee Envoy Adapter

[imesh]

Envoy is an open-source edge and service proxy designed for cloud native applications. Apigee Envoy Adapter can be used for intercepting traffic sent through Envoy and applying security and capturing analytics using Apigee Edge (Edge Cloud, OPDK, Hybrid).

In such deployments Envoy proxy could raise HTTP 403 error due to many reasons. In this article I have listed seven reasons for raising this error.

1) API product is not enabled for the specific environment:

API request:

curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 12 Jan 2021 08:18:08 GMT
server: envoy
RBAC: access denied

Apigee Envoy Adapter logs:

2021-01-12T08:18:08.124Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
2021-01-12T08:18:08.124Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: 7mQIG...
2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:125        using api key from request
2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
r)(0xc0001a0600), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"}, 
Expires:time.Time{wall:0x0, ext:63746037188, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
string{""}, APIKey:"7mQIG..."}
2021-01-12T08:18:08.589Z        DEBUG   product/manager.go:89   Authorizing request:
  products: [ENVOY-PRODUCT-1]
  scopes: []
  operation: GET /echo
  target: httpbin:8080
  - product: ENVOY-PRODUCT-1
    not found

In this scenario, API product was not enabled for the specific environment. To resolve this issue enable API product for the specific environment via the Apigee Edge UI, wait until API products are refreshed in Apigee Envoy Adapter, and send another request.

2) Missing target service URI path in API Product:

API request:

curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo1
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 12 Jan 2021 08:09:02 GMT
server: envoy
RBAC: access denied

Apigee Envoy Adapter logs:

2021-01-12T08:09:02.604Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:125        using api key from request
2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"}, 
Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
string{""}, APIKey:"7mQIG..."}
2021-01-12T08:09:02.605Z        DEBUG   product/manager.go:89   Authorizing request:
  products: [ENVOY-PRODUCT-1]
  scopes: []
  operation: GET /echo1
  target: httpbin:8080
  - product: ENVOY-PRODUCT-1
    no path: /echo1
2021-01-12T08:09:02.605Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)

In this scenario API request URI /echo1 was not found in the given API product. Add required API request URI to the relevant API product, wait until API products are refreshed in Apigee Envoy Adapter, and send another request.

3) The hostname given in the client API request is not found in the API product:

API request:

curl -i -H "x-api-key: $API_KEY" http://httpbin1:8080/echo 
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 12 Jan 2021 08:12:05 GMT
server: envoy
RBAC: access denied

Apigee Envoy Adapter logs:

2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:125        using api key from request
2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"}, 
Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
string{""}, APIKey:"7mQIG..."}
2021-01-12T08:12:06.019Z        DEBUG   product/manager.go:89   Authorizing request:
  products: [ENVOY-PRODUCT-1]
  scopes: []
  operation: GET /echo
  target: httpbin1:8080
  - product: ENVOY-PRODUCT-1
    no targets: httpbin1:8080
2021-01-12T08:12:06.020Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)

In this scenario hostname httpbin1:8080 was not found in the given API product. Add required hostname to the relevant API product via the UI under the section Apigee remote service targets or by adding a custom attribute with name "apigee-remote-service-targets", wait until API products are refreshed in Apigee Envoy Adapter, and send another request.

4) The API key is not passed in the "x-api-key" HTTP header:

API request:

curl -i http://httpbin:8080/echo
HTTP/1.1 403 Forbidden
date: Tue, 12 Jan 2021 08:20:30 GMT
server: envoy
content-length: 0

Apigee Envoy Adapter logs:

2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:98 Authenticate: key: , claims: map[string]interface {}(nil)
2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:159        Authenticate error: &auth.Context{Context:(*server.Handler)
(0xc0001a0600), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, 
loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [missing authentication]
2021-01-12T08:20:31.461Z        DEBUG   server/authorization.go:205     sending denied: UNAUTHENTICATED
2021-01-12T08:20:32.448Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header
: :authority

In this scenario, an API key was not sent in the API request using "x-api-key" HTTP header.

5) The API key passed as part of the request is invalid:

API request:

curl -i -H "x-api-key: $INVALID_API_KEY" http://httpbin:8080/echo
HTTP/1.1 403 Forbidden
date: Tue, 12 Jan 2021 08:26:12 GMT
server: envoy
content-length: 0

Apigee Envoy Adapter logs:

2021-01-12T08:26:12.275Z        DEBUG   auth/auth.go:98 Authenticate: key: , claims: map[string]interface {}(nil)
2021-01-12T08:26:12.275Z        DEBUG   auth/auth.go:159        Authenticate error: &auth.Context{Context:(*server.Handler)
(0xc0001e00c0), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, 
loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [missing authentication]
2021-01-12T08:26:12.275Z        DEBUG   server/authorization.go:205     sending denied: UNAUTHENTICATED
2021-01-12T08:26:12.526Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header
: :authority

In this scenario, the API key which was sent in the API request using "x-api-key" HTTP header was not valid.

6) Apigee Envoy Adapter is unable to communicate with the remote-service API proxy:

API request:

curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 12 Jan 2021 08:32:49 GMT
server: envoy
RBAC: access denied

Apigee Envoy Adapter logs:

2021-01-12T08:29:06.499Z        DEBUG   product/manager.go:188  retrieving products from: https://foo/remote-service/products
2021-01-12T08:29:06.505Z        ERROR   product/manager.go:164  Error retrieving products: Get "https://foo/remote-service/pro
ducts": dial tcp: lookup foo on 169.254.169.254:53: no such host
github.com/apigee/apigee-remote-service-golib/product.(*manager).start.func1
        /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/product/manager.go:164
github.com/apigee/apigee-remote-service-golib/util.(*Looper).Run
        /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:87
github.com/apigee/apigee-remote-service-golib/util.(*Looper).Start.func1
        /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:59

2021-01-12T08:32:49.369Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
2021-01-12T08:32:49.369Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: 7mQIG...
2021-01-12T08:32:49.561Z        DEBUG   auth/auth.go:159        Authenticate error: &auth.Context{Context:(*server.Handler)(0x
c0001c2600), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, loc:(*
time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [permission denied]
2021-01-12T08:32:49.561Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)

In this scenario, Apigee Envoy Adapter was unable to communicate with remote-service API proxy. This could occur if remote-service API proxy is not deployed in the specific environment or if there are network connectivity issues between Apigee Envoy Adapter and Apigee Edge environment.

7) Envoy proxy is unable to communicate with Apigee Envoy Adapter:

API request:

curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo
HTTP/1.1 403 Forbidden
date: Tue, 12 Jan 2021 08:34:58 GMT
server: envoy
content-length: 0

Apigee Envoy Adapter logs:

----[This issue was recreated by stopping Apigee Envoy Adapter. Hence, no logs available]----

This issue could occur if Envoy proxy is unable to communicate with Apigee Envoy Adapter. In such scenarios there will be no logs seen on Apigee Envoy Adapter.

[guycrets]

Imesh, thanks for this and your other great article about Apigee Envoy Adapter. Have been trying to get it working on OpenShift, but not yet working... One of the things I notice is that you switched to port 8080. So you've moved away from default httpbin.yaml. If I make progress on the Openshift side, will share it. KR, Guy Crets

[imesh]

@guycrets I am so glad to hear that two articles that I wrote so far on Apigee Envoy Adapter helped. Please create a question in the community and let us know the exact issue you are facing on OpenShift. We will do our very best to analyze and help you to get it solved.

Regarding the port, yes, I did few minor changes to the Envoy proxy configuration which was generated by the Envoy Adapter CLI. I did not mention much about the configuration here as I wanted to specifically focus on the errors and their root causes. Anyway, I will share those in another article. Thanks!