mTLS and Truststore

[srs-api]

I have mTLS enabled on one of our virtual host. The virtual host is pointing to a reference rather than directly to the truststore itself.

We have added couple of the client certificates to the the truststore. The question I have is if a new client comes in and we want to add their certificate and chain to the truststore , can we add to the existing one or would we need to create a new truststore all over again ? 

The reason I ask is because it seems that we have to create a new truststore when we have to renew expired certficates. Is this true only for renewal process or needs to happen when we add certificates to existign truststore?

[dchiesa1]

You should create a new truststore, and pivot the reference to point to that new truststore. 

Also - new clients shouldn't necessarily require a new trust chain.  If you use the same CA to issue certs to clients, then the same trusted root should work.  Right?  It's only if your new client has a cert from a new root CA , that you need to add to the truststore (or actually, to create a new truststore). 

[srs-api]

Thanks @dchiesa1   We are troubleshooting an mtls issue. Getting "400 The SSL certificate error". We looked at wireshark logs, compared the chain, certificate common name/issuer and also created a new truststore. But still getting the same error. Any thoughts on how we can find the issue? and we are using Apigee edge (public one)

[dchiesa1]

Yes, 

I don't like wireshark as a first step for diagnosing TLS handshake issues. 

In my experience the most common issue is a misconfigured peer - either the client or the server has a truststore or keystore that does not have the required settings, has an incorrect name, has an expired certificate, incomplete chain, etc. 

A good way to start exploring that is with openssl using the s_client command. 

This is a good overview.

The basic command is

openssl s_client -connect ${HOST}:${PORT}

You will need to ctrl-C out of that to end the command.

If your server uses SNI, then you need also to specify the server name:

openssl s_client -connect ${HOST}:${PORT} -servername ${HOST}

 You can also try these commands on hosts available from badssl.com , to see the results for various misconfigured systems.  Old ciphers, old protocols, wrong hostname, expired certs, etc. And then compare the results you see for your endpoint, against those results. This might give you some clues as to what is wrong. 

[API-Evangelist]

Often times we see this operational issue with setup of vHost with 2-way ssl or adding a new client cert..when ever we import a new certificate you will do below.

openssl s_client -connect {host}:{port}

Check for "Acceptable client certificate CA names" does your client cert showup after adding in the UI? If it doesn't then you simply restart router & re-check.(assuming you have proper setup) & most of the times it will resolve the issue.

Some good troubleshooting steps:

https://docs.apigee.com/api-platform/troubleshoot/runtime/400-ssl-certificate-error

& reference for 2-way ssl setup

https://docs.apigee.com/api-platform/fundamentals/configuring-virtual-hosts#definingavirtualhostfort...

[srs-api]

Is this restart needed on Apigee edge? Is the truststore cached?

[API-Evangelist]

Regular operational activity which involves edge-router to be restarted.

@dchiesa1..May be an item to engineering (well known operational issue )

[dchiesa1]

Yes, correct.  Unless you use the Reference, in which case, I believe no router restart is required.

[dchiesa1]

If you use the truststore reference (ref:// syntax), no restart required.

If you do n ot use the truststore reference, yes, you must restart the router.

[srs-api]

we are using the truststore reference as you have mentioned above. The behavior we are seeing is that we need to create a new truststore reference and point to it in the virtual host for it to take effect. Hence I was wondering if the truststore is getting cached even though it is used as a reference?