Istio mTLs not working after apigee-edge intergration
I had two service deployed to the Istio service mesh(version 1.0.4) . Service A configured to Ingress gateway for the user to call from outside the mesh, Service B which cal be called by only services inside the mesh through mTLS.
But after Apigee-Istio integration the call from Service A to Service B throws 403/Forbidden error.
Is this a bug or is there any config changes to be done.
Is this a bug or is there any config changes to be done.
You could also look into the logs of the ingress controller and look into why is a 403 being returned.
Otherwise, you need to provide a lot more details :
You said
- Service A to Service B call fails with a 403.
- Service A to Service B call is within the container cluster/mesh.
If(1) and (2) are true , how and where did Apigee gateway get involved? Is Service A calling Service B via apigee?
If yes, where is apigee installed? If apigee is installed outside this cluster then service B isn't a cluster service network call, it's actually coming from outside.In such a case service B, would need to be exposed on the ingress as well.
Additionally, what kind of authentication is this? Are you propagating tokens between services? Are you authenticating at each service?
You need to provide a lot more details about the layout of these services with apigee involved.
Thank you for the reply @Rahul M
Service A and B are part of the same Mesh cluster and same namespace too.
Service A is exposed on ingress and apigee adapter is configured on it. The call to Service A from outside the mesh is working with providing proper Auth Token from Apigee website.
But Service B is not configured with ingress , only Service A calls Service B which is in the same namespace. This was working fine earlier before apigee adapter was configured but after the configuration the call from Service A to B throws 403 error.
The call is simple from outside I call Service A which in turns calls Service B through a rest template but providing the url and NO-AUTH (i.e http:// called-service:8091/callme/ping) .
I have added the service graph below
------------------------------
---------------------------------
Error log from ingress is
{insertId: "13ojurcg1m48rcp" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"
receiveTimestamp: "2018-12-17T15:49:42.849463841Z
resource: {…}
severity: "INFO"
textPayload: "[2018-12-17T15:49:36.856Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 50 47 "10.128.0.9" "PostmanRuntime/7.4.0" "ad9f6e2c-9de3-9447-82c2-28b540ab5599" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "
timestamp: "2018-12-17T15:49:38Z"
{insertId: "1bqearxg1j8m9p6" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"
receiveTimestamp: "2018-12-17T15:50:02.855632832Z"
resource: {…}
severity: "INFO"
textPayload: "[2018-12-17T15:49:55.669Z] "GET /caller/pingHTTP/1.1" 403 - 0 75 5 5 "10.128.0.9" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" "5ce32548-3f6a-90b1-8178-1d37aa6017f1" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "
timestamp: "2018-12-17T15:49:58Z"
}
{insertId: "8brw4bfslt476" labels: {…}
logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"
receiveTimestamp: "2018-12-17T15:50:12.856362073Z"
resource: {…}
severity: "INFO"
textPayload: "[2018-12-17T15:50:02.832Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 32 31 "10.128.0.9" "PostmanRuntime/7.4.0" "91a94180-ea57-9261-a869-e3ec50a71ab7" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "
timestamp: "2018-12-17T15:50:08Z"
}
-------------------------------------------------------------------------------
Pod log for - istio-proxy
Service A called from outside mesh- [2018-12-17T16:33:05.671Z] "GET /caller/checkHTTP/1.1" 200 - 0 22 7 4 "10.128.0.9" "PostmanRuntime/7.4.0" "fcc47668-ba8b-9e80-8360-b673c155e4a3" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0
Service A called from outside mesh to call Service B -
[2018-12-17T16:33:43.063Z] "GET /callme/pingHTTP/1.1" 403 - 0 75 5 5 "-" "Java/1.8.0_181" "326e0e1c-1a9c-9485-a04b-105631779570" "called-service:8091" "10.40.1.29:8091" outbound|8091||called-service.default.svc.cluster.local - 10.43.250.126:8091 10.40.2.13:36892 [2018-12-17T16:33:43.055Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 22 18 "10.128.0.9" "PostmanRuntime/7.4.0" "33832b03-82d9-9fb9-b57b-e8b80cea1904" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0
Best Answer
The reason is because of the rules.yaml for apigee-rule which had
apiVersion: config.istio.io/v1alpha2kind: rulemetadata:name: apigee-rulenamespace: istio-systemspec:match: context.reporter.kind == "inbound"&& destination.namespace == "default"actions:-handler: apigee-handler.apigee.istio-systeminstances:- apigee.authorization- apigee.analytics
which states that all inbound traffic in the namespace default is to be authorized.
On changing the rule appropriately handles the token based as well as mtls.
Now we are getting somewhere.
I hope that Service A is not using a Service B with a URL like (i.e http:// called-service:8091/callme/ping). Since both services are within the cluster, they should rely on DNS resolution and not a name and a port.
Please check that.
Looking at the logs, it seems like your there were three requests, two returned 500 and one returned 403 - why did the response code change? what did you try?
Your Service B logs should tell you more.
Finally check your routing configuration (IP Tables/IPVS configuration/whatever you use) to see that the forwarding rules point to actual pod IP's.
If nothing else works, you can try doing a tcpdump on the node interface (if you have access) or the container eth0 interface and see why is the traffic being terminated that way.
The reason for three logs is because at Service A I haven't handled the exceptions so that throws 500, Service B is the one which is throwing 403 .
Actually I am calling Service B from Service a through a URL like http:// called-service:8091/callme/ping.
But this approach was working earlier before the Apigee gateway adapter integration, so is the adapter restricting it now ?
But now let me try the approach you had suggested .
@ Rahul M You mentioned to use DNS resolution and not by service name and port, what is that exactly, How do I call it that way.
I just have the service names and the port they are running on. To access shouldn't I just call service name and port ?
What is the container cluster you are using under istio? I mean "Rely on DNS resolution" and use the logical service name inside the cluster. for every container you install, the DNS gets an entry. If you can tell me the cluster you are using, i can tell you the command.
I am using a GKE(google kubernetes environment) cluster
Where you telling about the below service names ?
kubectl get svc --namespace=default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE called-service ClusterIP 10.43.250.126 <none> 8091/TCP 5d callerservice ClusterIP 10.43.247.132 <none> 8090/TCP 5d
Yes, assuming that both services are in the default namespace.
If all your underlying wiring is correct, you should be able to look up Called-service with the URL http://called-service.default.svc.cluster.local:8091 within the cluster
Also, what does called-service logs say?
From the looks of it, it doesn't seem to be an apigee problem - and this may not be the right forum for the question.
