SSL handshake failure for mTLS northbound connection with Apigee evaluation account

[martinjtlive]

SSL handshake failure for mTLS northbound connection with Apigee evaluation account

Hello,

I have set up an Apigee evaluation account. Within it I am using the default proxy endpoint (https://34.98.100.102.nip.io/hello-world) and target endpoint (https://mocktarget.apigee.net/user). I have entered the related xml elements to test mTLS. Here is a copy paste of my config. below
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TargetEndpoint name="default">
<PreFlow name="PreFlow">
<Request/>
<Response/>
</PreFlow>
<Flows/>
<PostFlow name="PostFlow">
<Request/>
<Response/>
</PostFlow>
<HTTPTargetConnection>
<URL>https://mocktarget.apigee.net/user</URL>
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://KS</KeyStore>
<KeyAlias>ServerCertAndKey</KeyAlias>
<TrustStore>ref://TS</TrustStore>
</SSLInfo>
</HTTPTargetConnection>
</TargetEndpoint>

To test mTLS, I plan to make a GET request to https://34.98.100.102.nip.io/hello-world via Postman.
Being a northbound connection, my understanding is the proxy endpoint is the server and Postman is the client. Thus using OpenSSL (specifically this link - https://blog.devolutions.net/2020/07/tutorial-how-to-generate-secure-self-signed-server-and-client-c...), I created certificates.
I created the CA key and certificate. When asked for details, I entered 'https://34.98.100.102.nip.io' as Common name.
Then I created the server key, server CSR and server certificate. This was followed by client key, client csr, client certificate.
The common name specified in both server and client csr is https://34.98.100.102.nip.io. Now in Apigee, I created 2 key stores. 1 key store has server certificate(in .pem format) and key. The other key store, I named as TS for Trust Store. It only have client certificate.
In Postman, in the certificates section, I added the server certificate(in .pem format) in the CA section. The client certificate and key in related section.

Now, when I do a GET on https://34.98.100.102.nip.io/hello-world, I get the following error:
{"fault":{"faultstring":"SSL Handshake failed sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","detail":{"errorcode":"messaging.adaptors.http.flow.SslHandshakeFailed"}}}

Can you please advise me on why I am getting this error and equally important is how to overcome this?

Could it be mTLS is not allowed for an evaluation account?
The OpenSSL page that I used to create the certificates indicated that the key is generated as a 256-bit private key over an elliptic curve, which is the industry standard. Could this be the issue? If so, is there another site that explains how to create certificates and keys for client and server?

Should the common name in the CSRs be different for CA, server and client? Should it be my apigee account hostname that is found here: https://apigee.google.com/organizations/orange-calm-234143/envgroups. It is something like this: orange-calm-234143-eval.apigee.net. I changed the hostname for privacy reasons.

Apigee team and users, your help would be greatly appreciated in my learning of Apigee. Please advise. Thank you.

[dchiesa1]

First, I'm not clear if you are using Apigee Edge or Apigee X.

You have a couple problems.

The error message you showed

SSL Handshake failed sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

... refers to the target. This is not relating to the connection between Postman and Apigee. It relates to the connection between Apigee and the upstream system, in this case, the mocktarget endpoint. Apigee is a proxy. Instead of a client connecting to an upstream system, the client connects to Apigee and Apigee connects to the upstream system. There are two connections. Each may have a TLS context, but those contexts will be different.

"Unable to find valid certification path" means , Apigee is trying to validate the target's credentials (= x509 certificate), and is unable to find a validation path. In other words, Apigee is unable to find a trusted certificate of a party that signed the target's certificate. Which suggests to me that postman was able to connect inbound to Apigee successfully.

One possible reason for the failure on the target is that you are using ref://TS for your TrustStore in the TargetEndpoint configuration, and you have no reference. When you use a ref:// prefix, then you must specify a TrustStore REFERENCE. Not the TrustStore directly, but a reference to the TrustStore. Your message didn't say anything about creating the reference. Conversely you did say you created a TrustStore named "TS". That suggests to me that you are not using a reference. If you have a targetendpoint that refers to a reference that does not exist, that might result in the "unable to find valid certification path". If so, remove the ref:// prefix from that element.

If you ARE using a reference, then... you need to make sure the referred-to TrustStore contains the cert for the CA that signed the cert for mocktarget.apigee.net. According to my check, that is GlobalSign Root CA.

$ openssl s_client -connect mocktarget.apigee.net:443 -servername mocktarget.apigee.net
CONNECTED(00000005)
depth=3 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1D4
verify return:1
depth=0 CN = mocktarget.apigee.net
verify return:1
---

This is the cert Apigee must trust, in order to trust mocktarget.apigee.net.

There will be a second problem. You have configured your TargetEndpoint to use 2-way TLS (ClientAuthEnabled = true). As far as I know, the mocktarget endpoint does not suport 2-way TLS. It uses 1-way TLS. So a 2-way handshake will never succeed.

Keep in mind that the TLS configuration between client and Apigee is distinct from that between Apigee and the upstream. You can configure Apigee to participate in a 2-way TLS handshake with the client (postman), and then a 1-way handshake with the upstream, if you like. Or vice versa. Just because you use 2-way TLS for the inbound (proxy, aka northbound) endpoint, does not mean you must use 2-way TLS for the outbound (target, aka southbound) endpoint.

Should the common name in the CSRs be different for CA, server and client?

Yes. The idea in X.509 is that each entity has its own certificate, with a unique common name. This is not particular to Apigee, of course. This is how TLS works. It sounds to me that you are experimenting with TLS in Apigee, and you're not quite clear on the theory of TLS in general. You may want to check out an overview of how TLS works. Apigee just does regular TLS things - it works exactly as every other TLS-compliant system works. Only the terminology may be slightly different, though most systems use terms like "keystore" and "truststore". So if you understand TLS in general, you will understand how it works in Apigee Edge.

If you're not clear on this TLS stuff in Apigee, then I suggest googling for past Q&A here on this forum. I've explained it multiple times before. For example, Here is an older answer re: 2-way TLS in Apigee Edge.

If you are using Apigee X, be aware that the configuration for inbound TLS is different from the analogous configuration in Apigee Edge. In Apigee X, you don't use the VirtualHost concept to configure 2-way TLS . (You didn't show any virtualhosts, but you said "northbound" and "truststore and keystore" and VirtualHost is how you configure the northbound connection in Apigee Edge.). In Apigee X, you can refer here for the steps to configure 2-way TLS on the inbound / northbound connection.

[martinjtlive]

Hi Dino,

I greatly appreciate your response. Please give me time to review it in detail and get back to you with responses. 

Again, thank you so much for detailed response. Sincerely, Martin

[martinjtlive]

Hi Dino,

>> First, I'm not clear if you are using Apigee Edge or Apigee X.

I believe I am using Apigee X. I say this because I created an evaluation account last week only and per this link - https://www.googlecloudcommunity.com/gc/Apigee/Apigee-X-vs-Apigee-Edge-both-supported-by-Google-what..., Apigee Edge is no longer for sale. I assume an evaluation account for Apigee Edge has also been discontinued as of last week or much earlier.

Further more, when I go to my landing page - https://apigee.google.com/landing, I dont see the word 'Edge' suffixed to the Apigee name anywhere on the page. For that matter, I dont see 'X' suffixed to Apigee anywhere on the page also. But my assessment is that I am using Apigee X.

>> ... refers to the target. This is not relating to the connection between Postman and Apigee. It relates to the connection between Apigee and the upstream system, in this case, the mocktarget endpoint. Apigee is a proxy.

Thank you for explaining this. My aim is to set up mTLS between my client (Postman) and the Apigee proxy. Is it allowed for an evaluation account?
Do you think if I add the following tags into the Proxy endpoint section and not the target endpoint section, it would work?

I'm considering adding the following xml into the proxy endpoint section:
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://KS</KeyStore>
<KeyAlias>ServerCertAndKey</KeyAlias>
<TrustStore>ref://TS</TrustStore>
</SSLInfo>

Location where to add the above into:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<APIProxy revision="2" name="hello-world">
<DisplayName>Hello World</DisplayName>
<Description>An example API proxy.</Description>
<CreatedAt>1656514305081</CreatedAt>
<LastModifiedAt>1656589140113</LastModifiedAt>
<BasePaths>/hello-world</BasePaths>
<ProxyEndpoints>
<ProxyEndpoint>default</ProxyEndpoint>
<!-- add the above here, perhaps, Dino? -->
</ProxyEndpoints>
<TargetEndpoints>
<TargetEndpoint>default</TargetEndpoint>
</TargetEndpoints>
</APIProxy>

If the above is valid and could work, my next concern is how do I get the cert and key for my proxy endpoint - https://34.98.100.102.nip.io/. Can I just generate the cert and key off openSSL using '34.98.100.102.nip.io' as the CN in the CSR? Or do I need to get it from the Apigee team as they assigned the test proxy to me. Please advise.

With regard to your comments on references, I do have references. Here is a screenshot

The strange thing is I cannot find the 'Trust Store' within https://apigee.google.com/landing. Where is the Trust store?
In my original post, I did the following: I went to Environment followed by TLS Keystores. There I created 2 Keystores of names: myKeyStore(Alias ServerCertAndKey); TrustStore (Alias TS). Please let me know if this is the wrong way to set up the Trust store with client's CA certificate. Following is screenshot:

If I am attempting to establish mTLS between Postman client and Apigee Proxy endpoint, then would I need to upload the CA cert that signed my client certificate into the Trust Store in Apigee X? I assume so. Please correct me if wrong.

Thank you very much for your previous reply. Can ypu please help with my questions above? It will be greatly appreciated Dino.

[dchiesa1 · Accepted solution]

I'm considering adding the following xml into the proxy endpoint section:
<SSLInfo>
<Enabled>true</Enabled>
<ClientAuthEnabled>true</ClientAuthEnabled>
<KeyStore>ref://KS</KeyStore>
<KeyAlias>ServerCertAndKey</KeyAlias>
<TrustStore>ref://TS</TrustStore>
</SSLInfo>

That won't work. In Apigee X you must follow the instructions I mentioned in the prior response. ( "In Apigee X, you can refer here for the steps to configure 2-way TLS on the inbound / northbound connection." )

As mentioned in that article, Apigee will not act as the TLS termination point in this case.  So you will not configure keys and certs in Apigee for this purpose. Check the article for full details.