Invalid certificate chain. Please check certificat...

drew · 03-31-2022 11:13 PM

I'm getting this error while attempting to enable Client Authorization for 2 way TLS. I can find the offending certificate in the trust store but I can't determine what could be wrong with it. Are there any more details that can be obtained from Apigee or instructions on how to validate certs?

dchiesa1

"Invalid cert chain" implies that the chain of trust cannot be terminated.

The way TLS works, there are "peers" on either end of the conversation. We can call them "client" and "server" to indicate the initiator of the connection, and the respondent, but from the point of view of TLS, they are peers. TLS allows the respondent to authenticate itself (aka "1 way TLS"), or it allows both peers to authenticate ("2-way TLS") . 1-way TLS is the common way of using TLS on websites: the browser initiates a connection to a server, the server then authenticates to the client, and if the client (browser) trusts that server, then the browser displays the website content.

OK, that's high level. How does one peer decide whether the authentication of the other peer is "good" - how does a peer decide to trust or not? This is based on x509 certificates, which are signed. In the simple case, the to-be-authenticated peer (the server in the 1-way case) sends its certificate to the challenging peer (client/browser in the 1-way case); the challenging peer then checks whether it trusts that certificate. It does this by checking in its truststore, to see if the signature on the other peer's cert can be verified with a cert in the client's truststore. I trust Bob. If I connect with Alice and Alice presents to me some evidence that Bob trusts Alice, then I can trust Alice.

It's not quite that simple because the x509 cert model is based on signature chains. I trust Bob. If I connect with Alice, and Alice presents to me evidence that Ruchika trusts her, and Ruchika also presents evidence that Bob trusts Ruchika, then.... by transitive trust I can trust Alice. It works like that with adjudicating trust in TLS with X509 certs. Party A can present a cert for itself signed by B, as well as a cert for B signed by C, as well as a cert for C signed by D, and if I trust D, then I can trust A. Theoretically, signing chains for X.509 certificates can be unlimited in length, though in practice they typically involve 3 or 4 links.

The key point here is that the peer that wants to prove its identity - in the 1-way case it's the server - must provide the cert chain to the other peer. All certs except the final "root" certificate. I'll say that again: the to-be-authenticated peer needs to send a chain of certificates to the peer that is checking its authentication. Using the openssl tool allows us to see that for any website. For example if I connect to www.reddit.com, I can see the cert chain it sends me:

$ openssl s_client  -connect www.reddit.com:443 -servername www.reddit.com 
CONNECTED(00000006)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = US, ST = CALIFORNIA, L = SAN FRANCISCO, O = Reddit Inc., CN = *.reddit.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=CALIFORNIA/L=SAN FRANCISCO/O=Reddit Inc./CN=*.reddit.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
...

Interpreting that, I see that the site sends a "wildcard" cert for *.reddit.com, which is signed by DigiCert (CN = DigiCert TLS RSA SHA256 2020 CA1). The site also sends the cert for DigiCert (CN = DigiCert TLS RSA SHA256 2020 CA1), which itself is signed by the DigiCert root CA (CN = DigiCert Global Root CA). The line with s: indicates subject, i: indicates issuer. So the cert chain looks good. the subject of cert 0 is signed by DigiCert TLS RSA SHA256 2020 CA1, and that is the subject of cert 1, which is issued by DigiCert Global Root CA.

A client (like a web browser) that wants to interact with www.reddit.com securely needs to validate that cert chain. The client must explicitly trust the root CA . Another way to say that: the cert for the root CA must be present in the Truststore.

TLS peers that want to prove their identities (the server in the 1-way TLS case) generally store that chain of certs in their Keystores, along with their key.

I would guess that "Invalid cert chain" means that one of the following is true:

there is a cert in the chain that is missing; in other words the peer is not sending a complete chain, OR
there is a cert in the chain that is expired, or not yet valid, or revoked, etc. OR
the certificate for the root CA is not present in the truststore.

A peer that checks authentication of an opposing peer (a browser in the case of 1-way TLS for websites) needs only the cert for the root CA in its Truststore, in order to validate the signing chain for a set of certificates. A peer that wants to prove its identity (a server in the case of 1-way TLS for websites) needs the certificate chain in its Keystore, in order to support transmitting that chain to the client.

Two-way TLS works basically the same way, except both peers need to send cert chains, and both peers validate trust via their own truststores.

OK, now let's translate to 2-way TLS with Apigee. In this case Apigee must validate the trust on the chain of certificates that the client is sending. So, you need to check

that the client is sending all the required certs, and that they comprise a chain. Eg, each cert has an issuer, and the next cert in the chain has a subject which is the issuer of the prior cert.
that each cert in that chain is valid, not expired, and not revoked.
that the truststore on the Apigee side has the cert for the root CA, which itself is not expired.

Check all of those things. In my experience, with 2-way TLS, the most common error is that the client itself is not configured properly, to send its full chain to the Apigee endpoint. Often this means the client-side keystore does not contain the full chain of certs. Start there.

drew

Thank you for that detailed explanation! I have some more questions but I should start by stating that I was successful in solving this issue by installing the complete certificate chain of the client certificate into my trust store.

I should have been a bit more clear in my original question I got the above not when attempting to connect to the virtual host using 2-way TLS but when trying to enable 2-way TLS on a virtual host. I got into this pickel because it seems that Apigee runs a type of validation on the trust store when 2-wat TLS is enabled but it's not run as new certs are installed. Basically, the following steps got me into this state:

create a trust store for client certs
select said trust store for 2-way TLS for a virtual host
enable 2-way TLS (AKA Client Authorization) on the virtual host
install certificates into the trust store
test 2-way TLS with clients and it works
disable 2-way TLS on the virtual host
attempt to enable 2-way TLS on the virtual host

At this point we get the "Invalid certificate chain. Please check certificates in TrustStore" error and we are unable to re-enable the 2-way TLS unless we remove the offending certificate.

Also, it seems that oddly for some of the clients I work with I can install only a certificate while with others I require the entire certificate chain.

So my questions are:

What is the preferred procedure one should follow when adding new certs to a trust store to avoid the above scenario?
Any idea why sometimes the cert chain is required and sometimes it's not?

dchiesa1

I have some more questions but I should start by stating that I was successful in solving this issue by installing the complete certificate chain of the client certificate into my trust store.

Glad to hear you sorted it out!

that oddly for some of the clients I work with I can install only a certificate while with others I require the entire certificate chain. ....
Any idea why sometimes the cert chain is required and sometimes it's not?

This would occur if some of the client applications are not configured to send their entire cert chain during TLS negotiation. In other words, what you describe would be expected if some of the clients were incorrectly configured. I would advise you to be careful about your solution. It's fine if you want to help out your client. By adding non-root certificates into your truststore, maybe even the full chain, you are telling Apigee to implicitly trust those certificates. That works, but... it means more exposure for you. For example if an intermediate CA is compromised, it means any new cert signed by that CA might be bogus, but you're trusting it implicitly. The general guidance is to install only root CAs (not the entire chain) in the truststore, and configure TLS peers to transmit their certificate chains (by placing the right certs in their KEYstore).

What is the preferred procedure one should follow when adding new certs to a trust store to avoid the above scenario?

As for steps 1 through 7, it seems like step 7 just reverses step 6? I understand that from an admin point of view, these things ought to be reversible. But ... especially in Apigee Edge, there is a need to restart some of the things in the chain to make the TLS handshake work properly. And that may happen laziily.

a best practice therefore, is not to update existing truststores with new certs etc, and then enable/disable TLS. If you're going to be modifying TLS setups, then you should use REFERENCES to truststores. https://docs.apigee.com/api-platform/system-administration/working-references

Quote from that page:

Apigee strongly recommends that you use references to keystores and truststores in a virtual host and do not use names directly.

drew

The general guidance is to install only root CAs (not the entire chain) in the truststore, and configure TLS peers to transmit their certificate chains (by placing the right certs in their KEYstore).

Are you recommending we install the root cert as a separate entry in the trust store or install the cert and the root cert together in 1 file without the intermediate cert?

a best practice therefore, is not to update existing truststores with new certs etc, and then enable/disable TLS. If you're going to be modifying TLS setups, then you should use REFERENCES to truststores.

I'm not clear on what the recommendation is here. We are using references to link the virtual host to the trust store and we trigger the Invalid certificate chain error when attempting to enable 2-way TLS while an invalid cert is in the trust store. If I follow the steps above up to step 5 then I get no errors. It seems that this procedure (1-5) allows me to add invalid certs to the trust store in use because Apigee only validates the cert chains while attempting to enable the 2-way TLS. To be more specific in the UX enabling Client Authorization as shown below:

It seems like there should be a procedure for adding a new client certificate to a truststore in use that validates that the new certificate is acceptable and keeps enforcement of 2-way TLS on. Other than cycling the 2-way TLS setting I don't see one in the Apigee admin interface.

dchiesa1

Are you recommending we install the root cert as a separate entry in the trust store or install the cert and the root cert together in 1 file without the intermediate cert?

Install certificates only for root CAs in the trust store. Certificated for client apps and for intermediate CAs do not belong in the truststore. Should be unnecessary if the client app is properly configured.

It seems like there should be a procedure for adding a new client certificate to a truststore in use that validates that the new certificate is acceptable and keeps enforcement of 2-way TLS on.

That's a fair request. I don't know if there is a limitation there. If so, you may wish to file an enhancement request with Apigee support.

It seems that this procedure (1-5) allows me to add invalid certs to the trust store in use

Can you clarify what you mean by "invalid certs"? Expired certificates? If not that, then what makes them invalid?

I understand you're not completely satisfied with the way the Truststores work. I'm trying to clarify just what problem I can help you solve here. Maybe there are multiple. If you are observing a bug in Apigee behavior, then the Apigee support engineers can help you log that, find workarounds, and request a fix. If you are unclear how things are intended to work, I might be able to explain.

drew

Install certificates only for root CAs in the trust store. Certificated for client apps and for intermediate CAs do not belong in the truststore. Should be unnecessary if the client app is properly configured.

Maybe we're mincing words but I thought the truststore setup for 2-way TLS needed to have the client's certificate loaded into it so that it only accepts connection requests from the clients presenting a cert in the truststore. If I install only the root certificates aren't I opening myself up for connections from any certificate signed by that root cert?

It seems that this procedure (1-5) allows me to add invalid certs to the trust store in use
Can you clarify what you mean by "invalid certs"? Expired certificates? If not that, then what makes them invalid?
I understand you're not completely satisfied with the way the Truststores work, but I'm tryint to clarify just what problem we are solving here. Maybe there are multiple. If you are observing a bug in Apigee behavior, then the Apigee support engineers can help you log that, find workarounds, and request a fix.

By "invalid" I mean a cert that when installed into a trust store triggers the Invalid certificate chain when attempting to enable 2-way TLS. It seems that the best way to handle this currently is to test the certificate using the procedure I outlined in the production truststore in use or in a test truststore. I was hoping there would be another alternative, like an openssl command I could run, but that procedure can work fine for us. We only need to add client certs once every month or 2.

dchiesa1

Maybe we're mincing words but I thought the truststore setup for 2-way TLS needed to have the client's certificate loaded into it so that it only accepts connection requests from the clients presenting a cert in the truststore.

If you load the Truststore with the client's cert that means you want to explicitly trust only that client's cert. That will "work" for that client. In my experience, though, organizations don't want to trust a single client, but rather, a set of clients. In the happy path, those clients all have certs signed by the same CA. Which means I need to load in the CA's cert into the trust store, and then yes, Apigee will trust all the certs signed by that CA. This is typically how X.509 truststores work. It's not special to Apigee.

If I install only the root certificates aren't I opening myself up for connections from any certificate signed by that root cert?

That's the goal. To have a more scalable way to manage many-to-many trust, x.509 uses the chain-of-trust mechanism. I trust Bob. If Bob asserts that he trusts Alice, then I can also trust Alice, implicitly. I don't need to be specifically introduced to Alice.

drew

I think I see some of our confusion. My organization actually does want to trust a limited set of specific certs from a relatively small number of companies. Thus we only install those certs into our trust store.

All the companies we work with obtain certs from their preferred signing authority. For some we can install the cert without the chain and it works fine however for others we require the entire certificate chain. This is what initiated my original question as I can't find a concrete explanation as to why some are fine while others generate the error above.

dchiesa1

I see, thanks for that, I am clearer now.

For some we can install the cert without the chain and it works fine however for others we require the entire certificate chain. This is what initiated my original question as I can't find a concrete explanation as to why some are fine while others generate the error above.

I think the issue is probably still on the client side. The way it works: when the client (or any peer) authenticates, it will send its entire configured chain of certificates. The correct configuration is that the certificates must be in order, so that cert0 identifies the client, the cert1 belongs to the CA that signed cert0, cert2 belongs to the intermediate CA that signed cert1, and so on, until the root of the chain, aka the root certificate. The client (really, any peer authenticating via TLS) need not send the root cert in the normal case, because in the normal case the root cert will be present in the truststore of the server, and so it's explicitly trusted and can be used to verify the signatures all the way up the chain. [By convention the cert of the root CA is "self-signed"; it is signed by itself, and the signature can be verified using the signed cert.]

In your case, you don't want to do that chaining thing. Any certificate you load into the Truststore will be explicitly trusted. It need not be a "cert for a root CA" which is self-signed. The Apigee runtime will treat any certificate in the Truststore as....um..... trusted!

As for why you need the entire chain in the truststore for some clients and, for others you need only the single cert.... I am unsure, but it could be that the client in the former case is not configured properly and is sending only the root certificate, or some intermediate certificate, as opposed to its own certificate along with the full chain. You will be able to see the CN, I, and S on the cert within the Apigee trace if you enable PropagateTlsInformation on the vhost. There will be context variables holding the information from the cert that the client used. (You may need to insert an AssignMessage policy to read those variables, in order for them to appear in trace.) Consult the documentation on this vhost property to learn more.

drew

The thing is I get the above error not when a client connects to my vhost but when I attempt to enable 2-way TLS. All I do is install the certs in a truststore and turn it on and the error is displayed. As shown below:

dchiesa1

I'm sorry I'm out of ideas. I suggest you contact the Apigee support desk!

drew

no worries. Thanks for the input.

Invalid certificate chain. Please check certificates in TrustStore: