Istio mTLs not working after apigee-edge intergration

I had two service deployed to the Istio service mesh(version 1.0.4) . Service A configured to Ingress gateway for the user to call from outside the mesh, Service B which cal be called by only services inside the mesh through mTLS.

But after Apigee-Istio integration the call from Service A to Service B throws 403/Forbidden error.

Is this a bug or is there any config changes to be done.

Is this a bug or is there any config changes to be done.

Solved Solved
0 14 786
1 ACCEPTED SOLUTION

The reason is because of the rules.yaml for apigee-rule which had

apiVersion: config.istio.io/v1alpha2kind: rulemetadata:name: apigee-rulenamespace: istio-systemspec:match: context.reporter.kind == "inbound"&& destination.namespace == "default"actions:-handler: apigee-handler.apigee.istio-systeminstances:- apigee.authorization- apigee.analytics

which states that all inbound traffic in the namespace default is to be authorized.

On changing the rule appropriately handles the token based as well as mtls.

View solution in original post

14 REPLIES 14

rmishra
Participant V

@Vignesh Shanmugam

You could also look into the logs of the ingress controller and look into why is a 403 being returned.

Otherwise, you need to provide a lot more details :

You said

  1. Service A to Service B call fails with a 403.
  2. Service A to Service B call is within the container cluster/mesh.

If(1) and (2) are true , how and where did Apigee gateway get involved? Is Service A calling Service B via apigee?

If yes, where is apigee installed? If apigee is installed outside this cluster then service B isn't a cluster service network call, it's actually coming from outside.In such a case service B, would need to be exposed on the ingress as well.

Additionally, what kind of authentication is this? Are you propagating tokens between services? Are you authenticating at each service?

You need to provide a lot more details about the layout of these services with apigee involved.

Thank you for the reply @Rahul M

Service A and B are part of the same Mesh cluster and same namespace too.

Service A is exposed on ingress and apigee adapter is configured on it. The call to Service A from outside the mesh is working with providing proper Auth Token from Apigee website.

But Service B is not configured with ingress , only Service A calls Service B which is in the same namespace. This was working fine earlier before apigee adapter was configured but after the configuration the call from Service A to B throws 403 error.

The call is simple from outside I call Service A which in turns calls Service B through a rest template but providing the url and NO-AUTH (i.e http:// called-service:8091/callme/ping) .

I have added the service graph below

------------------------------7867-capture.jpg---------------------------------

Error log from ingress is

{insertId: "13ojurcg1m48rcp" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:49:42.849463841Z

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:49:36.856Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 50 47 "10.128.0.9" "PostmanRuntime/7.4.0" "ad9f6e2c-9de3-9447-82c2-28b540ab5599" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:49:38Z"

{insertId: "1bqearxg1j8m9p6" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:50:02.855632832Z"

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:49:55.669Z] "GET /caller/pingHTTP/1.1" 403 - 0 75 5 5 "10.128.0.9" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" "5ce32548-3f6a-90b1-8178-1d37aa6017f1" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:49:58Z"

}

{insertId: "8brw4bfslt476" labels: {…}

logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:50:12.856362073Z"

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:50:02.832Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 32 31 "10.128.0.9" "PostmanRuntime/7.4.0" "91a94180-ea57-9261-a869-e3ec50a71ab7" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:50:08Z"

}

-------------------------------------------------------------------------------

Pod log for - istio-proxy

Service A called from outside mesh- [2018-12-17T16:33:05.671Z] "GET /caller/checkHTTP/1.1" 200 - 0 22 7 4 "10.128.0.9" "PostmanRuntime/7.4.0" "fcc47668-ba8b-9e80-8360-b673c155e4a3" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0

Service A called from outside mesh to call Service B -

[2018-12-17T16:33:43.063Z] "GET /callme/pingHTTP/1.1" 403 - 0 75 5 5 "-" "Java/1.8.0_181" "326e0e1c-1a9c-9485-a04b-105631779570" "called-service:8091" "10.40.1.29:8091" outbound|8091||called-service.default.svc.cluster.local - 10.43.250.126:8091 10.40.2.13:36892
[2018-12-17T16:33:43.055Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 22 18 "10.128.0.9" "PostmanRuntime/7.4.0" "33832b03-82d9-9fb9-b57b-e8b80cea1904" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0

rmishra
Participant V

Now we are getting somewhere.

I hope that Service A is not using a Service B with a URL like (i.e http:// called-service:8091/callme/ping). Since both services are within the cluster, they should rely on DNS resolution and not a name and a port.

Please check that.

Looking at the logs, it seems like your there were three requests, two returned 500 and one returned 403 - why did the response code change? what did you try?

Your Service B logs should tell you more.

Finally check your routing configuration (IP Tables/IPVS configuration/whatever you use) to see that the forwarding rules point to actual pod IP's.

If nothing else works, you can try doing a tcpdump on the node interface (if you have access) or the container eth0 interface and see why is the traffic being terminated that way.

The reason for three logs is because at Service A I haven't handled the exceptions so that throws 500, Service B is the one which is throwing 403 .

Actually I am calling Service B from Service a through a URL like http:// called-service:8091/callme/ping.

But this approach was working earlier before the Apigee gateway adapter integration, so is the adapter restricting it now ?

But now let me try the approach you had suggested .

@ Rahul M You mentioned to use DNS resolution and not by service name and port, what is that exactly, How do I call it that way.

I just have the service names and the port they are running on. To access shouldn't I just call service name and port ?

What is the container cluster you are using under istio? I mean "Rely on DNS resolution" and use the logical service name inside the cluster. for every container you install, the DNS gets an entry. If you can tell me the cluster you are using, i can tell you the command.

I am using a GKE(google kubernetes environment) cluster

Rahul M

Where you telling about the below service names ?


kubectl get svc --namespace=default

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
called-service   ClusterIP   10.43.250.126   <none>        8091/TCP   5d
callerservice    ClusterIP   10.43.247.132   <none>        8090/TCP   5d

Yes, assuming that both services are in the default namespace.

If all your underlying wiring is correct, you should be able to look up Called-service with the URL http://called-service.default.svc.cluster.local:8091 within the cluster

Also, what does called-service logs say?

From the looks of it, it doesn't seem to be an apigee problem - and this may not be the right forum for the question.

I tried the URL http://called-service within the cluster(no port required), it din't work. It is not accessible with out the port number.


This problem occurred after Apigee adapter integration it was working fine without the integration. That's why thought it was the forum to raise this questions.

Edited answer above ^

And are you sure the invocation to "called-service" is somehow not being intercepted by the Apigee adapter which is forcing an authenticate? 403 is weird, if called-service has no auth involved

The call reached the istio-proxy of the called-service.

Thats why even I am confused why 403 is thrown, apigee is at the mixer and for the ingress, not sure why the service to service interaction is getting interrupted .

The log from the called-service removing Apigee-adapter intergation:

[2018-12-18T14:58:37.565Z] "GET /callme/pingHTTP/1.1" 200 - 0 29 14 11 "-" "Java/1.8.0_181" "11aa0885-67ae-9ccf-a460-4addb66faf61" "called-service:8091" "127.0.0.1:8091" inbound|8091||called-service.default.svc.cluster.local - 10.36.1.17:8091 10.36.2.10:42788


The logs from the called-service Istio-proxy after Apigee-Adapter integration:

[2018-12-17T20:50:43.284Z] "GET /callme/pingHTTP/1.1" 403 - 0 75 6 - "-" "Java/1.8.0_181" "4a8353bb-e448-9145-9d86-15e1dd9e5c0f" "called-service:8091" "-" - - 10.40.2.24:8091 10.40.1.47:54292

The reason is because of the rules.yaml for apigee-rule which had

apiVersion: config.istio.io/v1alpha2kind: rulemetadata:name: apigee-rulenamespace: istio-systemspec:match: context.reporter.kind == "inbound"&& destination.namespace == "default"actions:-handler: apigee-handler.apigee.istio-systeminstances:- apigee.authorization- apigee.analytics

which states that all inbound traffic in the namespace default is to be authorized.

On changing the rule appropriately handles the token based as well as mtls.