Mutual TLS between client to edge and edge to backend

[rmasanam]

Hi

We are doing mutual TLS from client to Edge and Edge to backend.

In case of Client => Edge mTLS, what is recommended approach for clients to get their client cert from? Can they get it from public CA? can they get it from our company CA? If getting it from our company CA is fine for client certs, then we can have our company CA chain in our trust store and the same trust store can be used to validate any client. Otherwise , for any new client certs that are signed by diff public CAs, we keep adding cert chains in our trust store. To avoid this, can we have client certs signed by our company CA?

In case of Edge=> backend mTLS, I have to store public cert and private key in keystore to validate itself to backend. How to get this public cert? is it public CA signed or Apigee provided our company specific client cert or our company CA signed cert? we just need one for any backend or we need separate edge certs for each backend?

Thank you

[dchiesa1 · Accepted solution]

TLS is based on a federated trust model. The way Apigee Edge does TLS is ... standard. It's the way any TLS peer does TLS. So it does not matter which CA issues your cert. Allowing a client to connect is just a matter of establishing trust for the CA that signs the client certificate.

In TLS, trust of the certificate of the inbound client request is dependent upon the certificates you have loaded into the Truststore. Wha?.... Certificates Truststore, keys...? You may know this but I'll describe it anyway...

Each cert is signed, and according to the X.509 certificate verification model ....a receiver will trust a certificate if the signer is trusted. And a receiver (like Apigee Edge) trusts the signer... if the certificate of the signer is in the TrustStore. It's actually just one step more complicated than that - because the cert of the signer may also be signed by someone else. Therefore a TLS participant traces the entire chain of signatures to the root. (a root certificate is self-signed). If the root CA is trusted (is present in the truststore), then all the other certs in the chain are trusted.

BTW, what's a certificate? It's just a digital document that binds a name, like "Dino Chiesa" to a public key. Putting a cert into a Truststore tells the entity that relies on the Truststore, "trust any party that presents a thing you can verify with this public key." And this system relies on public/private key encryption: the idea is that the entity with the name listed in the Cert holds the private key that matches the public key in the cert.

To make this work, configure the certs of the signing root CA into the truststore. Conversely, the keystore is what you use to store the key+cert of your Apigee Edge endpoint.

This language - "keystore" and "truststore" - is what Apigee Edge uses, and this language is also used by other systems. So if you read other external articles on setting up TLS, you may see the same terms. To review:

• The keystore is the thing that holds PRIVATE keys that are used to assert the identity of a participant in a TLS transaction. If your company is "example.com" then the keystore will hold a key + cert that asserts the hostname as "api.example.com" . When contacted and requested by a client, the endpoint (Apigee Edge in this case) will present this cert which says "I assert that I am api.example.com") to the remote client . Then the other participant (sometimes called "the other peer") must evaluate whether to trust the certificate that was presented.
• The truststore is the thing that holds certificates of CAs, Certificate Authorities. These are bodies that sign other certificates. Using these certs and following the signing chain as I described above, a participant establishing a TLS connection can determine whether to trust the certificate offered by the peer participant.

In mutual TLS, TLS is a mutual protocol, so both ends engage in this process. Both peers present a cert, and then both peers evaluate whether the other's cert should be trusted. Both peers need a keystore and a truststore in order to manage the artifacts necessary to do this negotiation as described above. Only if both peers agree to trust the other's cert is there a valid connection.

In 1-way TLS, only the client (caller) verifies the server's identity. This is what happens typically when a web browser connects with a website like https://www.example.com . The web endpoint presents the cert and the browser looks in its TRUSTSTORE to evaluate whether to trust the presented certificate. The browser doesn't need a keystore because the browser is not authenticating itself (identifying itself?) to the remote server. Only the server is presenting an authenticated identity.

When you use configure 1-way TLS from Apigee Edge to a backend system, it works the same way as the typical browser case. The Target in Apigee Edge in this case needs a truststore to be configured, but not a keystore. When you use 1-way TLS inbound to Apigee Edge, it works the same way. Apigee Edge presents a cert to the calling client, and the client is responsible for evaluating whether to trust that cert.

---

You want to use 2-way (mutual) TLS. According to your description of the situation, whether you allow your clients to use only company-issued certificates, or certs issued by third parties, is a business decision. Maybe with some influencer from your IT Security team. Some companies insist that clients use only certs that the company itself issues. Other companies let the clients use any cert issued by a trusted CA.

As for trusting certs issued by "public CAs", it's not too difficult to honor those, if you want to do so. The industry relies on the Mozilla public trusted CA list. For example Google uses that list in the Android OS. And you can use those certs too, in Apigee Edge.

There is a pretty cool public (free) service available at mkcert.org, which you can use to help you obtain certs for the root CAs in Mozilla's list. mkcert doesn't define the list, it just provides a service to allow anyone to extract cert for public root CAs in PEM format.

I have a tool, a script, that uses the output of mkcert to provision a truststore within Apigee Edge. It basically pulls PEM files from mkcert.org and then uploads them into a Truststore within your Apigee Edge environment. This would allow you to trust any client cert signed by a public root CA. Or to trust only the certs signed by GoDaddy. Or only those signed by Verisign or GoDaddy. Etc.

Give me a moment to update it and post it here.

EDIT: ok, here is the script: provisionTruststoreWithMkcert.js.

[rmasanam]

Thank you Dino. Can you recommend on below situation as well.

In case of Edge=> backend mTLS, I have to store public cert and private key in keystore to validate itself to backend. How to get this public cert? is it public CA signed or Apigee provided our company specific client cert or our company CA signed cert? we just need one for any backend or we need separate edge certs for each backend?

[dchiesa1]

sorry, I had missed that part of your question.

Here again, Apigee Edge is just acting as a TLS peer. So, how you obtain the cert is a business decision. Usually companies rely on a 3rd-party root CA - a well-known public root certificate authority like GeoTrust or GoDaddy etc - to issue new certs. OR, your company might have a relationship with some other party that itself has a relationship with a root CA, and that third party issues new certs for your company.

It doesn't matter, either way... the way TLS verification works at each peer is: the peer looks at the cert, and looks at who signed the cert. The TLS peer checks:

• is the signer a root CA?
  • if so, do I trust the root CA? YES / NO
  • if signer is not a root CA, get parent signer of signers cert. and loop back to first step.

Apigee does not issue certificates.

You generally do not need or want a separate cert for each backend Apigee Edge communicates to, on your behalf. Think of the cert as the thing that uniquely identifies and corresponds to that "system", where "system" is the Apigee Edge proxy. Generally you will want a distinct cert for each system.

If you have a website - get a cert for that.

If you have an intranet site - get a distinct unique cert for that.

if you have an outward facing API endpoint (including an Apigee Edge proxy) - get a distinct cert for that.

if that API endpoint communicates to other systems, you need a cert for that communication.

The real question is.. .should you use the same cert on the front side of the Apigee Edge proxy, as you use on the back side ? And that is a matter of opinion, I suppose. My opinion: I would use different certificates, only for flexibility's sake.

so:

1. get a cert (and key) for the front side of Apigee Edge (just as you would get a cert for your website). Put that in the keystore for your vhost.
2. get a cert and key for the south side of Apigee Edge, and put that in the keystore for the SSLInfo you use for your target.
3. install the appropriate certs into the various Truststores on each end.
4. ba-da-boom. Mutual TLS.

Does this help?

Keep in mind that if you use ServiceCallout or JavaScript/httpClient, you will need to use the same SSLInfo there, as you do for your target, in order to get mutual TLS for the "upstream".

[rmasanam]

Thank you Dino. This is really helpful. Can you provide the script that you use to upload to truststore

[dchiesa1]

yes, the I linked to the script in an edited version of my original response.

here's the script again:

provisionTruststoreWithMkcert.js

I should write an article on this tool, I think.

[rmasanam]

Do we need one cert per org (in case of edge => backend communication), or just one cert for all orgs. please advise what is recommended.

[dchiesa1]

The model is: You need to configure an SSLInfo for each target. The SSLInfo will reference a keystore and a truststore. Those are entities within an ENVIRONMENT, which is subsidiary to an org.

You can use the same cert in multiple environments, or even multiple orgs. They important part is to configure your certs in the keystore and truststore.

This language - "keystore" and "truststore" - is what Apigee Edge uses, and this language is also used by other systems. So if you read other external articles on setting up TLS, you may see the same terms.

• The keystore is the thing that holds PRIVATE keys that are used to assert the identity of a participant in a TLS transaction. If your company is "example.com" then the keystore will hold a key + cert that asserts the hostname as "api.example.com" . When contacted and requested by a client, the endpoint (Apigee Edge in this case) will present this cert which says "I assert that I am api.example.com") to the remote client . Then the other participant (sometimes called "the other peer") must evaluate whether to trust the certificate that was presented.
• The truststore is the thing that holds certificates of CAs, Certificate Authorities. These are bodies that sign other certificates. Using these certs and following the signing chain as I described above, a participant establishing a TLS connection can determine whether to trust the certificate offered by the peer participant.

In mutual TLS, TLS is a mutual protocol, so both ends engage in this process. Both peers present a cert, and then both peers evaluate whether the other's cert should be trusted. Both peers need a keystore and a truststore in order to manage the artifacts necessary to do this negotiation. Only if both peers agree to trust the other's cert is there a valid connection.

In 1-way TLS, only the client (caller) verifies the server's identity. This is what happens typically when a web browser connects with a website like https://www.example.com . The web endpoint presents the cert and the browser looks in its TRUSTSTORE to evaluate whether to trust the presented certificate. The browser doesn't need a keystore because the browser is not authenticating itself (identifying itself?) to the remote server. Only the server is presenting an authenticated identity.

When you use configure 1-way TLS from Apigee Edge to a backend system, it works the same way as the typical browser case. The Target in Apigee Edge in this case needs a truststore to be configured, but not a keystore. See? Makes sense?

[rmasanam]

keystore and truststore part I am aware, just wanted to check on how many certs needed as these certs cost us right. But I got the answer. Thank you.