How to configure client-side SSL between apigee edge and backend?

[cladius_fernand]

Currently my backend is employing white-listing to allow apigee's ip-addresses to access it and block out other traffic. However the proper solution would be to enable client-side ssl i.e. importing a certificate from the server onto the apigee layer. For achieving this the backend has been configured to only allow requests which provide the certificate. For eg: if I access my backend proxy from a browser it fails by default. For a successful request, I need to import the .p12 file in the browser. I've tried this out in FF and Chrome, and it works properly. I believe this is similar to:

curl --cert ./cert.pem --key ./key.pem <backendurl>

I tried following the steps mentioned at: http://apigee.com/docs/api-services/content/about-... to enable a similar connection from apigee edge to my backend but to no avail. To summarize I did the following steps:

1. I converted the .p12 certificate to .pem format using openssl and obtained 2 files (cert.pem and key.pem) [Since apigee does not accept the .p12 format.]
2. I created a keyStore jar with the cert.pem, key.pem and descriptor.properties file (with the standard folder structure).
3. I created a keyStore using the management api and uploaded the jar to it.
4. I created a trustStore using the management api and uploaded the cert.pem to it.
5. I modified the TargetServer which was being used in my api-proxy to use the above entities as:

{
  "host": "mybackend.com",
  "isEnabled": true,
  "name": "backendProxy",
  "port": 443,
  "sSLInfo": {
    "ciphers": [],
    "enabled": true,
    "clientAuthEnabled": true,
    "keyAlias": "keyAlias",
    "keyStore": "keyStore",
    "trustStore": "trustStore",
    "ignoreValidationErrors": true,      
    "protocols": []
  }
}

I have tried out different combinations for the parameters in the above TargetServer configuration i.e. enabled, clientAuthEnabled, etc. What am I doing wrong over here? Any pointers? For my scenario do I actually need keyStore and trustStore both or only one of them? Any pointers would be greatly appreciated.

[adas]

@Cladius Fernando If you want to do 1-way SSL then you don't need keystores. You can simply set enabled to true under SSLInfo, the rest is not required. If you want to do 2-way SSL then you do need them. You need the following:

• Create a keystore on Edge and upload the Edge cert and private key. This cert and private key is typically supplied by the backend server.
• Create a truststore on Edge that contains the cert and CA chain that you received from the backend server.
• Make sure you are uploading the corrects certs in the truststore with the CA chain.

[cladius_fernand]

Thanks @arghya das. A couple of queries, the scenario that I have described above, that qualifies as 2-way ssl. Is that correct?

For keystore you have mentioned - "the cert and key are typically supplied by the backend server". And for truststore you have said - "received from the backend server". So are we referring to the same cert.pem file in both keystore and truststore? (While keystore will also have key.pem)

Are there any samples for this besides the ones mentioned on the documentation page.

[adas]

@Cladius Fernando yes your scenario does qualify as a 2-way ssl case because you want mutual authentication. And no these certs are not the same. I will try and explain a bit about the keystore and truststore concepts and how they are used in the 2-way SSL handshake mechanism.

The keystore like you already know contains a jar file which consists of a pem file, which is either a CA signed certificate or a self-signed cert and the private key, which is also a pem file

The truststore would only contain the certs as PEM files but no private keys. When using truststore if you have a chain of certs, you must upload the individual PEM files.

For 2-way ssl, both the client (Apigee EDGE) and server (your backend) both maintain their own keystore with the cert and private key. The truststore contains trusted certificates stored on the client that are used when the client makes an secure connection to server endpoint that's ssl enabled. For two-way SSL, the contents of the truststore are used to validate the identity of the server's certificate being presented to the client. So the client cert must be something that the backend provides and you need to provision them into the EDGE truststore.

You can find more details in our docs site: http://apigee.com/docs/api-services/content/keysto...

I will check if I can find an existing community article that can help you.

[cladius_fernand]

Thanks for the response. I will try this out and reply back on this thread.

[cladius_fernand]

Thanks @arghya das for the info. Success is still eluding me. Steps followed by me after incorporating your pointers:

1. Converted the .p12 certificate (which works in the browser to allow me to access the backend) to .pem format using openssl and obtained 2 files (cert.pem and key.pem).
2. Created keystore.jar with the cert.pem, key.pem and descriptor.properties files (with the standard folder structure).
3. Created a keyStore using the management api and uploaded keystore.jar to it.
4. Downloaded my backend's certificate chain via the browser into separate .pem files - backend0.pem, backend1.pem and backend2.pem. (backend0.pem belongs to the CA and backend2.pem belongs to the backend server.)
5. Created 3 trustStores (trustStore0, trustStore1 and trustStore2) using the management api and uploaded backend0.pem, backend1.pem and backend2.pem to them correspondingly.
6. Modified the TargetServer which was being used in my api-proxy to use the above entities as:

{
 "host": "mybackend.com", 
 "isEnabled": true, 
 "name": "backendProxy", 
 "port": 443,
 "sSLInfo": {
   "keyStore": "keyStore",
   "keyAlias": "keyAlias",
   "trustStore": "trustStore2",
   "clientAuthEnabled": true,
   "enabled": true,
   "ignoreValidationErrors": true
 }
}

For the above steps could you please mention if each step is either - correct, incorrect or not-required? I am still facing the following error message:

{
  "fault": {
    "faultstring": "Unexpected EOF",
    "detail": {
      "errorcode": "messaging.adaptors.http.UnexpectedEOF"
    }
  }
}

Any pointers?

[adas]

Everything looks fine except Step 5. You don't need to create 3 truststores. You need to concatenate the certs into a single pem file:

When using a truststore, if you have a chain of certs, you must upload the chain as individual PEM files.

Once that's done just refer to that truststore in the SSLInfo.

[cladius_fernand]

Hi @arghya das. I tried that out as well. I created truststore3 with backend_chain.pem and mentioned it in the targetserver. However, I am still facing the same issue. I can see trustStores0, trustStores1, trustStores2 and trustStores3 under Edge Mgmt UI > Admin > SSL. I also see an entry for the keyStore. Any other pointers?

[cladius_fernand]

Hi @Floyd Jones, in light of what @arghya das has mentioned, shouldn't the note at keystores/truststores documentation be - "you must upload the chain as a single PEM file" instead of saying "as individual PEM files"? I have logged a comment on the page via bugherd as well.

[jonesfloyd]

@Stephen Gilson - Comment?

[sgilson]

Engineering is still validating that setup to see if that is the intended behavior.

Stephen

[sgilson]

Engineering has confirmed that when using a truststore, and you have a chain of certs, you can upload the chain as a single PEM files. The first cert is the certificate to be used for SSL - followed by the chain of certs, in order, to the CA certificate.

Stephen

[adas]

@Cladius Fernando Can you post the exact error that you see, when you make a runtime call. I would also suggest you enable debug logging on the MPs, restart the MPs and attach those logs (or send me via an email) so that we can look into it.

About the certificate chain thing, that line was indeed copied from the docs site. Also make sure you maintained the order of the certs like shown below. Here's the screenshot:

[cladius_fernand]

Hi @arghya das, our commercial org is on the cloud and as such I do not have access to the MPs. I have verified the order of the certificates (i.e. the CA cert is the one at the bottom). The error that I am encountering is as mentioned in my earlier comment:

{
  "fault": {
    "faultstring": "Unexpected EOF",
    "detail": {
      "errorcode": "messaging.adaptors.http.UnexpectedEOF"
    }
  }
}

Could you please share your email id? Btw, I have shared more details about the org on the apigee support portal - Case #903346.

[adevegowda · Accepted solution]

@Cladius Fernando

As informed in the support ticket, we had to do the following:

1. Modify target server definition to point to the truststore (truststore3) that contains the backend server's certificate chain

(as already mentioned by @arghya das, we need to have all the 3 certificates in one single .pem file)

2. Every time, we create a new keystore/truststore and upload the certificates, we have to restart the Message Processor(s) to get it effected.

With these two changes, I am able to see successful responses for your API calls. Please do verify this at your end.

[sgilson]

@AMAR DEVEGOWDA Since this is a long thread, I wanted to be clear about one thing:

The doc says "When using a truststore, if you have a chain of certs, you must upload the chain as individual PEM files."

But you are saying a truststore has to contain the certs in a single file. I thought that was only for a keystore?

Stephen

[#17]

@Birute Awasthi This seems like it might warrant a callout to update documentation or maybe a full blown article ?

[birute]

Agreed. @docs will look into possible documentation updates. @AMAR DEVEGOWDA @arghya das @Cladius Fernando - want take a stab at putting this into an article that summarizes the wisdom of this thread?

[sgilson]

Docs updated here and article posted here.

Stephen

[#20]

You guys are awesome 🙂

[#21]

@Cladius Fernando @arghya das

How do we get to know the value of the keyAlias?