Two-way TLS to the F5 Load Balancer

[#1]

Hi,

I've attempted to setup two way ssl on target servers and our F5 load balancer is not receiving the correct handshake.

I've used the SSL Test Tool on the datacentre truststore (public certs only, retrieved using firefox) with the apigee client cert + key and got "Received fatal alert: handshake_failure".

I was able to curl it locally successfully with the same cert.

Any ideas?

Thanks,
Dáire

[dchiesa1]

Can you show me the curl command you used? Did you use the Apigee Certs in the curl command?

2-way TLS requires that both peers, in your case both Apigee Edge and the F5 device, trust the other's certificate.

The tLS handshake can fail if either of the peers does not trust the others.

If you use curl to perform a 1-way TLS connection, then curl will verify/validate the cert of the remote peer (F5), against its cacerts file. This of course does not send a client cert to F5, for F5 to validate. So if you use curl in this way, it's an interesting test, in that it proves that the certs on the F5 device are trusted by curl. But it does not prove the other required trust relationship.

To perform a 2-way TLS connection with curl, you must pass the client cert and key. It looks like this:

curl  -E /path/to/client/certificate --key /path/to/client/private/key https://server/url/path

Is that what you did?

And can you tell me how you got the contents of the certificate and key file ?

---

Aside from all this testing, you need to make sure the F5 device is set up with a truststore, and set up to verify the cert belonging to the remote peer (n this case Apigee Edge), which is acting as a client to the F5.

does this help?

One more thing - you need to make sure you've configured the F5 device with a virtualserver - to be a Layer 7 proxy. Often F5 BigIP is used as a Layer 4 device, and in that case it won't terminate TLS. In that case the task for TLS termination is left to the thing behind the F5 device, which may be your app.

[#3]

Hi,

I can confirm that the curl command to test the connection was

curl -v --key apigee.client.key --cert apigee.client.crt  https://server/url/path

I've checked with network team and we terminate SSL at the F5 load balancer. The Virtual Server Type is set up as a "Standard" type.

I've moved the config out of a target server and received the response in Trace

Received fatal alert: handshake_failure

My config is as below:

    <HTTPTargetConnection>
        <URL>https://my.server.com</URL>
        <Path>/</Path>
        <SSLInfo>
            <Enabled>true</Enabled>
            <ClientAuthEnabled>true</ClientAuthEnabled>
            <KeyStore>ref://apigeeClient</KeyStore>
            <KeyAlias>apigee-client</KeyAlias>
            <TrustStore>ref://dataCentreKeystore</TrustStore>
            <Ciphers>
                <Cipher>TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384</Cipher>
                <Cipher>TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384</Cipher>
                <Cipher>TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256</Cipher>
                <Cipher>TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256</Cipher>
            </Ciphers>
        </SSLInfo>
    </HTTPTargetConnection>

[dchiesa1]

ok good. Thanks for confirming it's truly 2-way TLS and the F5 device is terminating the TLS connection. The handshake failure error message isn't particularly helpful, unfortunately. I'd like Apigee Edge to deliver more context when that happens, but unfortunately that's not possible at the moment.

Now, the apigee.client.key - how did you obtain the contents of that file? I suppose you used a CSR and generated a cert. I guess it is in the apigee.client.crt file, can you confirm that?

And can you show me the metadata for that cert, as generated by something like this:

openssl x509 -inform pem -in cerfile.cer -noout -text 

Does the metadata look right to you?

Also , have you imported the signing cert into the truststore of the F5 virtualhost?

And can you confirm that you correctly imported THAT CERT into the keystore of Apigee Edge referenced by ref://apigeeClient ?

1. query the keystore ref to see the keystore
2. query the keystore and alias "apigee-client"
3. compare the metadata you see there with the metadata you get from openssl. Are they the same? They should be !

(All of this is standard TLS stuff)

After looking at the cert used by the client (in this case, either curl or Apigee Edge), you need to do the analogous checks on the other peer. Check the cert being used by the F5 device. Check that the metadata is as you expect. Check that the truststore ref used by Apigee edge points to the right truststore. Check the contents of the truststore to make sure that a signing chain for the cert used by the F5 device, can be validated through it.

[#5 · Accepted solution]

I figured it out in the end, extra newline character (additional blank line) in our cert and key files. Our F5 must be very sensitive.

[dchiesa1]

Ouch!

What a pain! TLS is complicated enough without that kind of pitfall.

That IS very sensitive. Normally the processor would trim the whitespace and newlines.