{ Community }
  • Academy
  • Docs
  • Developers
  • Resources
    • Community Articles
    • Apigee on GitHub
    • Code Samples
    • Videos & eBooks
    • Accelerator Methodology
  • Support
  • Ask a Question
  • Spaces
    • Product Announcements
    • General
    • Edge/API Management
    • Developer Portal (Drupal-based)
    • Developer Portal (Integrated)
    • API Design
    • APIM on Istio
    • Extensions
    • Business of APIs
    • Academy/Certification
    • Adapter for Envoy
    • Analytics
    • Events
    • Hybrid
    • Integration (AWS, PCF, Etc.)
    • Microgateway
    • Monetization
    • Private Cloud Deployment
    • 日本語コミュニティ
    • Insights
    • IoT Apigee Link
    • BaaS/Usergrid
    • BaaS Transition/Migration
    • Apigee-127
    • New Customers
    • Topics
    • Questions
    • Articles
    • Ideas
    • Leaderboard
    • Badges
  • Log in
  • Sign up

Get answers, ideas, and support from the Apigee Community

  • Home /
  • APIM on Istio /
avatar image
0
Question by Vignesh Shanmugam · Dec 12, 2018 at 07:56 PM · 257 Views api management

Istio mTLs not working after apigee-edge intergration

I had two service deployed to the Istio service mesh(version 1.0.4) . Service A configured to Ingress gateway for the user to call from outside the mesh, Service B which cal be called by only services inside the mesh through mTLS.

But after Apigee-Istio integration the call from Service A to Service B throws 403/Forbidden error.

Is this a bug or is there any config changes to be done.

Is this a bug or is there any config changes to be done.

Comment
Add comment Show 2
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Rahul M · Dec 16, 2018 at 10:39 PM 0
Link

@Vignesh Shanmugam

You could also look into the logs of the ingress controller and look into why is a 403 being returned.

Otherwise, you need to provide a lot more details :

You said

  1. Service A to Service B call fails with a 403.
  2. Service A to Service B call is within the container cluster/mesh.

If(1) and (2) are true , how and where did Apigee gateway get involved? Is Service A calling Service B via apigee?

If yes, where is apigee installed? If apigee is installed outside this cluster then service B isn't a cluster service network call, it's actually coming from outside.In such a case service B, would need to be exposed on the ingress as well.

Additionally, what kind of authentication is this? Are you propagating tokens between services? Are you authenticating at each service?

You need to provide a lot more details about the layout of these services with apigee involved.

avatar image Vignesh Shanmugam Rahul M · Dec 17, 2018 at 04:28 PM 0
Link

Thank you for the reply @Rahul M

Service A and B are part of the same Mesh cluster and same namespace too.

Service A is exposed on ingress and apigee adapter is configured on it. The call to Service A from outside the mesh is working with providing proper Auth Token from Apigee website.

But Service B is not configured with ingress , only Service A calls Service B which is in the same namespace. This was working fine earlier before apigee adapter was configured but after the configuration the call from Service A to B throws 403 error.

The call is simple from outside I call Service A which in turns calls Service B through a rest template but providing the url and NO-AUTH (i.e http:// called-service:8091/callme/ping) .

I have added the service graph below

---------------------------------------------------------------

Error log from ingress is

{insertId: "13ojurcg1m48rcp" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:49:42.849463841Z

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:49:36.856Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 50 47 "10.128.0.9" "PostmanRuntime/7.4.0" "ad9f6e2c-9de3-9447-82c2-28b540ab5599" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:49:38Z"

{insertId: "1bqearxg1j8m9p6" labels: {…} logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:50:02.855632832Z"

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:49:55.669Z] "GET /caller/pingHTTP/1.1" 403 - 0 75 5 5 "10.128.0.9" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" "5ce32548-3f6a-90b1-8178-1d37aa6017f1" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:49:58Z"

}

{insertId: "8brw4bfslt476" labels: {…}

logName: "projects/ibesx-esa-sandbox-00f7f67a/logs/istio-proxy"

receiveTimestamp: "2018-12-17T15:50:12.856362073Z"

resource: {…}

severity: "INFO"

textPayload: "[2018-12-17T15:50:02.832Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 32 31 "10.128.0.9" "PostmanRuntime/7.4.0" "91a94180-ea57-9261-a869-e3ec50a71ab7" "35.232.73.122" "10.40.2.13:8090" outbound|8090||callerservice.default.svc.cluster.local - 10.40.1.30:80 10.128.0.9:57873 "

timestamp: "2018-12-17T15:50:08Z"

}

-------------------------------------------------------------------------------

Pod log for - istio-proxy

Service A called from outside mesh- [2018-12-17T16:33:05.671Z] "GET /caller/checkHTTP/1.1" 200 - 0 22 7 4 "10.128.0.9" "PostmanRuntime/7.4.0" "fcc47668-ba8b-9e80-8360-b673c155e4a3" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0

Service A called from outside mesh to call Service B -

[2018-12-17T16:33:43.063Z] "GET /callme/pingHTTP/1.1" 403 - 0 75 5 5 "-" "Java/1.8.0_181" "326e0e1c-1a9c-9485-a04b-105631779570" "called-service:8091" "10.40.1.29:8091" outbound|8091||called-service.default.svc.cluster.local - 10.43.250.126:8091 10.40.2.13:36892
[2018-12-17T16:33:43.055Z] "GET /caller/pingHTTP/1.1" 500 - 0 137 22 18 "10.128.0.9" "PostmanRuntime/7.4.0" "33832b03-82d9-9fb9-b57b-e8b80cea1904" "35.232.73.122" "127.0.0.1:8090" inbound|8090||callerservice.default.svc.cluster.local - 10.40.2.13:8090 10.128.0.9:0

Close

2 Answers

  • Sort: 
avatar image
1
Best Answer

Answer by Vignesh Shanmugam · Dec 19, 2018 at 07:38 PM

The reason is because of the rules.yaml for apigee-rule which had

apiVersion: config.istio.io/v1alpha2kind: rulemetadata:name: apigee-rulenamespace: istio-systemspec:match: context.reporter.kind == "inbound"&& destination.namespace == "default"actions:-handler: apigee-handler.apigee.istio-systeminstances:- apigee.authorization- apigee.analytics

which states that all inbound traffic in the namespace default is to be authorized.

On changing the rule appropriately handles the token based as well as mtls.

Comment
Add comment · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image
0

Answer by Rahul M · Dec 17, 2018 at 04:48 PM

Now we are getting somewhere.

I hope that Service A is not using a Service B with a URL like (i.e http:// called-service:8091/callme/ping). Since both services are within the cluster, they should rely on DNS resolution and not a name and a port.

Please check that.

Looking at the logs, it seems like your there were three requests, two returned 500 and one returned 403 - why did the response code change? what did you try?

Your Service B logs should tell you more.

Finally check your routing configuration (IP Tables/IPVS configuration/whatever you use) to see that the forwarding rules point to actual pod IP's.

If nothing else works, you can try doing a tcpdump on the node interface (if you have access) or the container eth0 interface and see why is the traffic being terminated that way.

Comment
Add comment Show 10 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Vignesh Shanmugam · Dec 17, 2018 at 04:55 PM 0
Link

The reason for three logs is because at Service A I haven't handled the exceptions so that throws 500, Service B is the one which is throwing 403 .

Actually I am calling Service B from Service a through a URL like http:// called-service:8091/callme/ping.

But this approach was working earlier before the Apigee gateway adapter integration, so is the adapter restricting it now ?

But now let me try the approach you had suggested .

avatar image Vignesh Shanmugam · Dec 17, 2018 at 05:20 PM 0
Link

@ Rahul M You mentioned to use DNS resolution and not by service name and port, what is that exactly, How do I call it that way.

I just have the service names and the port they are running on. To access shouldn't I just call service name and port ?

avatar image Rahul M Vignesh Shanmugam · Dec 17, 2018 at 06:06 PM 0
Link

What is the container cluster you are using under istio? I mean "Rely on DNS resolution" and use the logical service name inside the cluster. for every container you install, the DNS gets an entry. If you can tell me the cluster you are using, i can tell you the command.

avatar image Vignesh Shanmugam Rahul M · Dec 17, 2018 at 07:23 PM 0
Link

I am using a GKE(google kubernetes environment) cluster

avatar image Vignesh Shanmugam Vignesh Shanmugam · Dec 17, 2018 at 08:09 PM 0
Link

Rahul M

Where you telling about the below service names ?


kubectl get svc --namespace=default

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
called-service   ClusterIP   10.43.250.126   <none>        8091/TCP   5d
callerservice    ClusterIP   10.43.247.132   <none>        8090/TCP   5d
avatar image Rahul M Vignesh Shanmugam · Dec 18, 2018 at 03:23 AM 0
Link

Yes, assuming that both services are in the default namespace.

If all your underlying wiring is correct, you should be able to look up Called-service with the URL http://called-service.default.svc.cluster.local:8091 within the cluster

Also, what does called-service logs say?

From the looks of it, it doesn't seem to be an apigee problem - and this may not be the right forum for the question.

Show more comments

Follow this Question

Answers Answers and Comments

64 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

What is recommended: Apigee's Adaptor for Istio or Apigee Edge Proxies? 1 Answer

  • Products
    • Edge - APIs
    • Insights - Big Data
    • Plans
  • Developers
    • Overview
    • Documentation
  • Resources
    • Overview
    • Blog
    • Apigee Institute
    • Academy
    • Documentation
  • Company
    • Overview
    • Press
    • Customers
    • Partners
    • Team
    • Events
    • Careers
    • Contact Us
  • Support
    • Support Overview
    • Documentation
    • Status
    • Edge Support Portal
    • Privacy Policy
    • Terms & Conditions
© 2021 Apigee Corp. All rights reserved. - Apigee Community Terms of Use - Powered by AnswerHub
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Post an idea
  • Spaces
  • Product Announcements
  • General
  • Edge/API Management
  • Developer Portal (Drupal-based)
  • Developer Portal (Integrated)
  • API Design
  • APIM on Istio
  • Extensions
  • Business of APIs
  • Academy/Certification
  • Adapter for Envoy
  • Analytics
  • Events
  • Hybrid
  • Integration (AWS, PCF, Etc.)
  • Microgateway
  • Monetization
  • Private Cloud Deployment
  • 日本語コミュニティ
  • Insights
  • IoT Apigee Link
  • BaaS/Usergrid
  • BaaS Transition/Migration
  • Apigee-127
  • New Customers
  • Explore
  • Topics
  • Questions
  • Articles
  • Ideas
  • Badges