SSL hand shake error

[lintu_cherian]

We have configured our API proxy to connect to an external API  and getting below handshake error .

Found same error from Message processor logs .

error      The Service is temporarily unavailable

type       ErrorPoint

state      TARGET_REQ_FLOW

error.class           com.apigee.errors.http.server.ServiceUnavailableException

error.cause         Received fatal alert: handshake_failure

Identifier             fault

{"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}

------------------------------------------------------------------------------------

We are getting proper response using curl from message processor server .

curl -v -X POST https://xxxxxxxxxxxxxxxxx/authenticate -H 'content-type: application/json' -d '{ "username": "user", "password":"xyzgff" }'

* About to connect() to xxxxx port 443 (#0)

*   Trying 83.xx.xx.xxx...

* Connected to xxxxxxxxxxxxxx (83.xxx.xx.xxx)port 443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using xxxxxxxxxxxxx

* Server certificate:

*       subject: CN=*.xxxxxxx

*       start date: Dec 04 00:00:00 2020 GMT

*       expire date: Dec 08 23:59:59 2021 GMT

*       common name: *.xxxxxxxx

*       issuer: CN=xxxxxxxxx

> POST /xxxx/authenticate HTTP/1.1

> User-Agent: curl/7.29.0

> Host: xxxxxxxxxx

> Accept: */*

> content-type: application/json

> Content-Length: 41

> 

* upload completely sent off: 41 out of 41 bytes

< HTTP/1.1 200 OK

{"data":{"token":"eyJhbGciOiJI.uWJ0a1wA6diThWTTwzVQf1IJ509VYYH6VgemUrUXD94","ttl":86400},"statusCode":0,"message":"success","timeStamp":"2021-07-25T06:41:13.229Z"}

---------------------------------

OPen SSL openssl s_client -connect [servername]:443  -- worked fine certificates got downloaded successfully .

We are getting SSL hand shake error only when trying through Apigee proxy , no  truststore were created as curl worked without certificates.

[Denis_KALITVI]

curl works from OS because curl uses OS truststore (../security/cacerts). 

If thi is mTLS connection > you have to configure TLS keystore in Apigee, create reference and use it in SSLINFO block of your target server. 

from your curl I see that this is regular TLS. can you show us how you configured the SSLInfo block on target server? What ciphers and TLS version this backend uses? 

[lintu_cherian]

Thanks for the response Denis . Please find the SSL info block  .

<HTTPTargetConnection>
<Properties/>
<SSLInfo>
<Enabled>true</Enabled>
<Protocols>
<Protocol>TLSv1.2</Protocol>
</Protocols>
</SSLInfo>
<URL>https://xxxxxxxx/authenticate</URL>
</HTTPTargetConnection>

Below are the cipher and TLS version observed from openssl command .

---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3621 bytes and written 452 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 3C676F45646C6CDC3BCEBD9ABC223CE85D053E40B518CAE0DC7825E57C87173C
Session-ID-ctx:
Master-Key: D1B372EBCD104698678A1F4F58E6FB7F5913CFF78F27CEA84570B8A9439FD9573979AC636B118EC6A3DC5A420724AFB1
Key-Arg : None

[Denis_KALITVI]

Well, if this is regular TLS (not mutual-tls, mTLS), you can use this target server config, because you using <Enabled>true</Enabled> tag, you asking from Apigee to validate the backend cert, (but somehow ny the docs you are using correct syntax...)  and you didn't provided any ref:// to trusted store, so: 

1. either use this config: 

    <HTTPTargetConnection>
        <Properties/>
        <URL>https://httpbin.org/get</URL>
    </HTTPTargetConnection>

it will works for regular https (TLS) without validating the backend certificates

or you need to include reference to your trust store like this:

  <HTTPTargetConnection>
    <SSLInfo>
      <Enabled>true</Enabled>
      <TrustStore>ref://myTrustStoreRef</TrustStore>
    </SSLInfo>
    <URL>https://myservice.com</URL>
  </HTTPTargetConnection>

 try them both, 

@dchiesa1 any thoughts? 

upd. in my env (opdk) all works just like this: 

    <HTTPTargetConnection>
        <Properties/>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
        <Protocols>
            <Protocol>TLSv1.2</Protocol>
        </Protocols>
        <URL>https://httpbin.org/get</URL>
    </HTTPTargetConnection>

can you check if the httpbin reference works for you? maybe hostname different from your endpoint? 

[lintu_cherian]

Hello Denis ,

For 1st and 3rd option I was getting same error as mentioned above .

For 2nd option where trust-store reference was provided below error found in MP logs .

onfigurationBean not available for Path : /organizations/nonproductioncrs/maskconfigs/default . Returning empty DebugMaskDataConfigurationBean
2021-07-25 14:40:31,879 SSL Compliance Test NIO Thread INFO SSL_COMPLIANCE - SSLComplianceTest.run() : Starting a test SSL connection to STATIC-TARGET[/organizations/nonproductioncrs/environments/tqcmajor/apiproxies/MockTest/revisions/1]
2021-07-25 14:40:31,881 SSL Compliance Test NIO Thread ERROR SSL_COMPLIANCE - SSLComplianceTest.run() : STATIC-TARGET[/organizations/nonproductioncrs/environments/tqcmajor/apiproxies/MockTest/revisions/1], SSLDetails[enabled:true, clientAuthEnabled:false, keyStore:null, keyAlias:null, trustStore:TRAPIAP_TS, crlStorenull, commonName:null, useWildCardMatch: false,ciphers: [], protocols: [TLSv1.1], ignoreValidations: true] has no Key store
2021-07-25 14:40:31,907 org:nonproductioncrs env:tqcmajor api:MockTest rev:1 messageid:xxxxxxx01.corp.du.ae-24312-13159-3 NIOThread@2 ERROR HTTP.CLIENT - HTTPClient$Context.handshakeFailed() : SSLClientChannel[Connected: Remote:83.110.55.111:443 Local:172.24.242.4:51560]@656 useCount=1 bytesRead=0 bytesWritten=0 age=25ms lastIO=25ms isOpen=true handshake failed, message: Received fatal alert: handshake_failure
2021-07-25 14:40:31,907 org:nonproductioncrs env:tqcmajor api:MockTest rev:1 messageid:meydvaew01.corp.du.ae-24312-13159-3 NIOThread@2 ERROR HTTP.CLIENT - HTTPClient$Context.handshakeFailed() : SSLInfo:
KeyStore:null KeyAlias:null TrustStore:java.security.KeyStore@12d2e3a4
2021-07-25 14:40:31,908 org:nonproductioncrs env:tqcmajor api:MockTest rev:1 messageid:meydvaew01.corp.du.ae-24312-13159-3 NIOThread@2 ERROR ADAPTORS.HTTP.FLOW - RequestWriteListener.onException() : RequestWriteListener.onException(HTTPRequest@7a46ef4)
javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at com.apigee.nio.SSLTransport.processUnwrap(SSLTransport.java:275)
at com.apigee.nio.SSLTransport.run(SSLTransport.java:151)
at com.apigee.nio.SSLTransport.process(SSLTransport.java:524)
at com.apigee.nio.ClientChannel.process(ClientChannel.java:290)
at com.apigee.nio.NIOSelector$SelectedIterator.findNext(NIOSelector.java:574)
at com.apigee.nio.NIOSelector$SelectedIterator.findNext(NIOSelector.java:546)
at com.apigee.nio.util.NonNullIterator.computeNext(NonNullIterator.java:21)
at com.apigee.nio.util.AbstractIterator.hasNext(AbstractIterator.java:47)
at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:329)
at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:319)
at com.apigee.nio.util.NonNullIterator.computeNext(NonNullIterator.java:21)
at com.apigee.nio.util.AbstractIterator.hasNext(AbstractIterator.java:47)
at com.apigee.nio.handlers.NIOThread.run(NIOThread.java:141)

-----------------------------------------

 also during TCP dump comparison between curl and through api  observed that the cipher suite(Cipher : ECDHE-RSA-AES256-GCM-SHA384) was missing when tested through  api . Do we have any option to force using missing cipher in API ,

[Denis_KALITVI]

Yes, you can force ciphers: 

   <Ciphers>
      <Cipher>TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384</Cipher>    
   </Ciphers>

Try and tell me if this helped your case. It is strange that this is happening...

From your error you don't have any certs (or wrong alias) so MP can't find the cert. 

Also see this part: 

SSL_COMPLIANCE - SSLComplianceTest.run() : STATIC-TARGET[/organizations/nonproductioncrs/environments/tqcmajor/apiproxies/MockTest/revisions/1], SSLDetails[enabled:true, clientAuthEnabled:false, keyStore:null, keyAlias:null, trustStore:TRAPIAP_TS, crlStorenull, commonName:null, useWildCardMatch: false,ciphers: [], protocols: [TLSv1.1], ignoreValidations: true] has no Key store

TLS 1.1 is probably not supported, but I'm not sure if this relevant to your case. As your CURL shows 1.2 and you enforce 1.2 on Apigee level. Anyway, try this TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 cipher and tell me how it is going. 

BTW - do you have another HTTPS endpoint to see if this is this specific endpoint problem? 

[lintu_cherian]

HTTPTargetConnection>
<Properties/>
<SSLInfo>
<Enabled>true</Enabled>
<Protocols>
<Protocol>TLSv1.2</Protocol>
</Protocols>
<Ciphers>
<Cipher>TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384</Cipher>
</Ciphers>
<ClientAuthEnabled>false</ClientAuthEnabled>
<IgnoreValidationErrors>true</IgnoreValidationErrors>
<TrustStore>TRAPIAP_TS</TrustStore>
</SSLInfo>
<URL>https://xxxxxx/authenticate</URL>
</HTTPTargetConnection>

And getting below error when tried to save .

[Denis_KALITVI]

Maybe this is the problem, this cipher not supported and your endpoint requires one

[lintu_cherian]

@dchiesa1  - Could you please look into this issue.

[dchiesa1]

I just tried this in the Apigee cloud, and this cipher works for me. I can save a proxy using the cipher you used. 

In my case, the echo endpoint I was using did not support that cipher, so an API request through the proxy did not succeed'; the TLS handshake failed. But that is not the problem you are describing anyway. 

I think that you are running OPDK, and the version of Java you have does not support the cipher you are asking for.  Either you have disabled that cipher in some way, or it is not enabled because of lack of strong crypto in your JVM. 

Maybe you can list the set of ciphers available on. your JVM with something like this: 

import java.security.SecureRandom;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocketFactory;

public class GetTLSCipherSuites {
  public static void main(String[] args) {
    try {
      SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
      sslContext.init(null, null, new SecureRandom());
      SSLSocketFactory socketFactory = sslContext.getSocketFactory();
      java.util.Arrays.asList(socketFactory.getSupportedCipherSuites()).stream()
          .forEach(System.out::println);
    } catch (java.lang.Exception exc1) {
      System.out.println("Exception:" + exc1.toString());
      exc1.printStackTrace();
    }
  }
}

...and see if the cipher you wish to use is actually included in the list. If not, then your JVM does not support it. 

Also I would ask, are you really certain you need to specify the exact cipher to use in TLS negotiation?  would it be ok to specify a set of them and allow the TLS handshake to proceed with one of the acceptable ciphers? 

For example you could specify this list: 

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384

[Denis_KALITVI]

I'm somehow sure that if you will not explicitly specify cipher(s), Apigee will try and communicate with cipher received from 'Server Hello', and this cipher might be not supported from reasons you wrote (java version, OS, etc.)

[dchiesa1]

If you do not explicitly specify ciphers, then the peers (client and server, if you like) negotiate the cipher to use based on what each of them supports. If you DO specify the list of acceptable ciphers, you're basically limiting the set of acceptable ciphers on the Apigee side, which constrains the negotiation that happens between the peers when establishing the TLS connection. Keep in mind that the partner peer also has a limited set of ciphers. For example according to TLSSled.sh, the Google Cloud Load Balancer in front of appengine supports only these TLSv1.2 ciphers:

ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA
ECDHE-RSA-AES256-SHA
AES128-GCM-SHA256
AES256-GCM-SHA384
AES128-SHA
AES256-SHA

These are not the same names as is used in the Java-based Apigee MP, but you get the idea:  It's a limited set. Notice: no ciphers in the ECDHE-ECDSA-* family!

The key is to limit things on the Apigee side, to (a) provide the minimum acceptable security (eg disable weak ciphers), and (b) allow maximum likelihood of a good cipher in the intersection of the set supported by the remote peer and the set supported by Apigee.

[lintu_cherian]

@dchiesa1  @Former Community Member - thanks for the guidance .

I have being testing without listing ciphers exclusively .When tried to add cipher apigee was giving error as not in supported list .

Also tried to exclusively add this cipher suite in message.properties file , but while restarting MP services changes are getting overwritten .

You have mentioned ""Notice: no ciphers in the ECDHE-ECDSA-* family!"" - did you mean there will be no option to add below cipher "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384".

[API-Evangelist]

Hello @lintu_cherian ,

Here is the very basic troubleshooting you will do.

1. You reported

error.cause Received fatal alert: handshake_failure

{"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}

Question: Is it 1-way ssl or you are trying to do 2-way ssl?

Just follow thru below document step by step ( you might have done but kindly re-visit & troubleshoot as you are very close to the systems.)

https://docs.apigee.com/api-platform/system-administration/configuring-ssl-edge-backend-service

For the error you reported could be a configuration issue (read thru this & take assistance from the network team to capture packets. Below should solve your issue for sure.

https://docs.apigee.com/api-platform/troubleshoot/runtime/ssl-handshake-failures

Few other tips for 503 (this may be un-related for you but good to know)

https://docs.apigee.com/api-platform/troubleshoot/runtime/503-service-unavailable

More tips:

Ciphers in your setup is incorrect - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 in apigee. We need to follow below

==

 Apigee accepts cipher suites only in the OpenSSL cipher strings format at both the virtual host and the Router. The OpenSSL ciphers manpage provides the SSL or TLS cipher suites from the relevant specification and their OpenSSL equivalents.

For example:

If you want to configure the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 cipher suite at the virtual host or the Apigee Router, then you need to identify the corresponding OpenSSL cipher string from the OpenSSL ciphers manpage. The OpenSSL cipher string for the cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is ECDHE-RSA-AES128-GCM-SHA256. So, you need to use the OpenSSL cipher string ECDHE-RSA-AES128-GCM-SHA256 while configuring the cipher suite in the virtual host or on the Apigee Router.

==

If you want to really solve just step back and follow step by step and troubleshoot if it doesn't work we can continue or  open a ticket for assistance..Some times it feels better with self resolution 🙂

Thankyou.

[lintu_cherian · Accepted solution]

Missed to update solution here .
This issue was fixed by enabling Java Cryptography Extension (JCE)

Below command to check if JCE is enabled or not

[apigwops@meylvaeap04 bin]$ ./jrunscript -e 'exit (javax.crypto.Cipher.getMaxAllowedKeyLength("RC5") >= 256 ? 0 : 1);'; echo $?
0 means enabled , 1 means not enabled

Below command was used to list the ciphers supported by jdk
java.util.Arrays.asList(javax.net.ssl.SSLServerSocketFactory.getDefault().getSupportedCipherSuites()).stream().forEach(print);

and below URL to follow to enable JCE
https://www.oracle.com/java/technologies/javase-jce-all-downloads.html