AWS Lambda + ServiceCallout: handshake_failure

Calling out to an AWS Lambda endpoint from a ServiceCallout returns:

Execution of ServiceCallout GetAccessToken failed. Reason: Received fatal alert: handshake_failure

ServiceCallout:

<ServiceCallout name="GetAccessToken">
    <DisplayName>GetToken</DisplayName>
    <Request variable="token-credentials-request"/>
    <HTTPTargetConnection>
        <URL>https:/xxxxxxxxxxx.amazonaws.com/token</URL>
        <SSLInfo>
            <Enabled>true</Enabled>
...

AssignMessage:

<AssignMessage name="AssignMessage-set-token-credentials">
    <AssignTo createNew="true" type="request">token-credentials-request</AssignTo>
    <Set>
        <QueryParams>
            <QueryParam name="username">{request.queryparam.username}</QueryParam>
            <QueryParam name="password">{request.queryparam.password}</QueryParam>
        </QueryParams>
        <Verb>GET</Verb>
    </Set>
    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
...

Any thoughts appreciated.

Michael McD.

FYI: the url when sent via a browser or postman returns json payload as expected as do non AWS Lambda URL endpoints.

0 10 3,108
10 REPLIES 10

Looks like it ought to work. I don't know the amazonaws token API, but I'm guessing you have that part sorted.

The "handshake failure" indicates that one side or the other is not trusting the TLS negotiation. The ServiceCallout is resolving the hostname (xxxxx.amazonaws.com) and is reaching it, but the next step in the connection, the TLS handshake, is failing.

What could be going wrong? I can think of a few things.

  1. Hostname mismatch.

    This is probably the most common thing I see.

    Point ssllabs.com to your amazonaws endpoint and interrogate the TLS/SSL settings. Don't worry - this is safe. SSLLabs uses only documented, standard protocol interfaces to evaluate your host. You may learn something in the results, like which cert is being presented. You said Lambda, so... maybe this is an AWS provided cert. In which case the cert will almost certainly match. But regardless which party (you, or Amazon) is providing the cert, check it. Make sure the host name in the cert matches the hostname you are using to connect in the URL specified in the HTTPTargetConnection.

  2. If xxxxxx.amazonaws.com uses a cert signed by a well-known CA, then the server authentication should work just fine. In the unlikely event that it does not, you can insert the trust chain for Amazon's cert into a TrustStore in Apigee Edge, and specify the TrustStore in the SSLInfo element.

    <SSLInfo>
      <Enabled>true</Enabled>
      <TrustStore>myTruststore</TrustStore>
      ...
    	

    Even if it seems unlikely that the ServiceCallout is not trusting the AWS cert (like if the cert is signed by someone famous, like Kanye), I suggest you try this anyway. Oh, and to retrieve the certs in a format you can use for the truststore, you can use openssl, as described by this superuser post. You need the full chain of certs, not just ONE. Then create and fill the truststore as described here. Modify your HTTPTargetConnection and test.

  3. Does the token endpoint require client authentication, too? (in other words, is it 2-way TLS?) In that case you will need to specify the appropriate settings in the SSLInfo element in that ServiceCallout.

    <SSLInfo>
      <Enabled>true</Enabled>
      ...
      <ClientAuthEnabled>true</ClientAuthEnabled>
      <!-- and KeyStore, and KeyAlias -->
    	

    And of course that Client stuff can be used with the TrustStore I described above.

    This stuff is all documented in the reference for the policy (Thanks to the Apigee docs team! you guys rock!)

  4. Another possibility?: the server may be rejecting the client because the client supports protocols the server does not consider to be secure. For example TLS v1. You can restrict the protocols the client (in this case, the ServiceCallout policy within Apigee Edge) uses to connect to the server using something like this:

    <SSLInfo>
      <Protocols>
        <!-- <Protocol>TLSv1</Protocol> -->
        <Protocol>TLSv1.2</Protocol>
      </Protocols>
    	

    I couldn't find a specific page on the SSLInfo configuration for outbound HTTPTargetConnections. But I think this page applies. It refers specifically to SSLInfo used within a virtual host, which is an inbound connection. But I think those parameters (Protocols, Ciphers) also apply to outbound (client) connections as specified within HTTPTargetConnection. I think, but not certain. We should improve the doc in this regard ( @Floyd Jones , what do you think?)

Sorry I can't provide any definitive answer. Just stuff to try.

When I have connected to AWS Lambda from within Apigee Edge, I've done so using a nodejs target running in Apigee Edge. It uses the AWS SDK for nodejs, and it works well. But what you are trying *should work also*, of course.

ps: I was kidding about Kanye. He doesn't sign certs.

Thanks for pointers...

  1. The AWS cert looks good (i.e. it's a Symantec Class 3 Secure Server CA - G4 issued cert)
  2. Tried setting the SSL/TLS protocol explicitly e.g. 1.2, 1.1 etc.

Still no joy.

I note that it's a wild card cert... that wouldn't have any bearing on the handshake would it?

In the mean time I'm going to switch to using a nodejs target. I'll post my findings on line here.

Thanks again,

Michael McD

Ps do you know of an example of an Edge NodeJS target using the Aws Sdk for NodeJS to call a Lambda function?

Please see the following

// labmdaBridge.js
// ------------------------------------------------------------------
//
// connect to Lambda functions...
//


var AWS     = require('aws-sdk'),
    express = require('express'),
    app     = express();


AWS.config.loadFromPath('./config.json');
var lambda  = new AWS.Lambda();


// Generic Send Error Function
function sendError(res, code, msg) {
  res.status(code);
  res.json({ error: msg });
}


var knownEntityTypes = [ 'Cart' ];


var entityToIdPropertyName = {
      'Cart' : 'cartId'
    };


function entityTypeIsKnown(entity) {
  return (knownEntityTypes.indexOf(entity) > -1);
}


// GET /Cart/aaaa-bbbb-cccd
app.get('/:entityType/:entityId', function(req, res) {
  try {
    // validate
    if (entityTypeIsKnown(req.params.entityType)) {
      var payload = {};
      payload[entityToIdPropertyName[req.params.entityType]] = req.params.entityId;
      if (req.query.filter) {
        payload.descriptionFilter = req.query.filter;
      }
      if (req.query.sortPrice) {
        payload.orderByPrice = true;
      }
      var params = {
            FunctionName: req.params.entityType,
            Payload: JSON.stringify( payload )
          };


      // invoke the first Lambda function
      lambda.invoke(params, function(e, data) {
        if (e) {
          console.log("Error: for lambda", e);
          sendError(res, 500, "Lambda Error - " + e.message);
          return ;
        }
        console.log("%s(%s): %s",
                    req.params.entityType, req.params.entityId, data.Payload);
        if (data.Payload == "null") {
          res.status(404).end();
          return;
        }
        if (data.Payload == "null") {
          res.status(404).end();
          return;
        }
        var payload = JSON.parse(data.Payload);
        if (payload.errorMessage || payload.errorType) {
          console.log("Error in lambda fn: ", data.Payload);
          payload.additionalNote = "the attached error was emitted by AWS lambda";
          sendError(res, 500, payload);
          return ;
        }


        // the first request has succeeded. Can now make a second ...
        var params2 = {
              FunctionName: 'AdditionalFunctionNameHere',
              Payload: data.Payload // or whatever you like 
            };


        // invoke the second Lambda function
        lambda.invoke(params2, function(e, data2) {
          if (e) {
            console.log("Error: for lambda", e);
            sendError(res, 500, "Lambda Error - " + e.message);
            return ;
          }


          console.log("Respinse: %s", JSON.stringify(data2.Payload));


          res.status(200);
          payload.secondPayload = data2.Payload;
          res.json(payload);
        });


      });
    }
    else {
      sendError(res, 404, "Unknown entity type");
    }
  }
  catch(e) {
    sendError(res, 500, "internal server error\n" + e.message);
  }
});

// catch and return 404
app.use(function(req, res, next) {
  res.status(404);
  res.json({ error: { code: 404.01, message: "Not found" }});
});

// start the server
var listener = app.listen(process.env.PORT || 7000, function() {
      console.log('lambda bridge up');
      console.log('endpoint: ' + JSON.stringify(lambda.endpoint));
    });

You need a config file that looks like:

{
"accessKeyId": "YOUR_KEY_HERE",
"secretAccessKey": "YOUR_SECRET_HERE",
"region": "us-east-1"
}

and I used this in my package.json

  "dependencies": {
    "aws-sdk": "^2.2.30",
    "express": "^4.13.3"
  }
  ...

Thanks for the ping, @Dino! We do have SSLInfo in HTTPTargetConnection covered in the API proxy configuration reference, but I've made additional tweaks/x-refs to (I hope) make it easier to find.

Thanks, Floyd!

I believe we need to get Service Name Indication enabled.

Which is a new feature/enhancement:Release notes.

Ahhhh, well that would apply for *inbound* connections to Apigee Edge, would it not?

Not sure about outbound requests to AWS Lambda. If SNI plays a role, then it is AWS that needs to implement it. I guess it's possible that AWS is implementing SNI And Apigee Edge is not, for that outbound request. Not sure.

Yeah I think your reasoning is sound... we're clutching at straws at the minute.

Also as it seems that we're not going to get SNI enabled anytime soon, I've raised this as a support issue.

Oh and thanks for the Node AWS SDK example.

Hi Michael,

I have an additional possibility. You mentioned SNI, and you may be right. I failed to consider the possibility that it is amazonaws.com that is supporting SNI.

Recently I researched a different problem and determined that Edge appears to be affected by a bug in Java8 that is related to SNI. You can read about it here.

The short story is that ServiceCallout is connecting to xxxx.amazonaws.com , which (according to my theory) is a server that uses SNI.

If my theory is correct, the Java library in Edge is not "doing the SNI client thing" properly, so Edge is getting an incorrect cert, or a cert that you don't expect.

Here's a way you can diagnose this a little further. Use openssl and interrogate the certs delivered by that xxxx.amazonaws.com host. I would do it but i don't have the hostname (and that's the way you want it, probably.) Here are the commands.

Request the cert using SNI:

openssl s_client  -connect xxxx.amazonaws.com:443 -servername xxxx.amazonaws.com

Request the cert using no SNI:

openssl s_client -connect xxxx.amazonaws.com:443

If you see two different certs, it indicates that the host is using SNI, and you are probably affected by the Apigee Edge bug APIRT-3770, which I described in that other comment.

Please let me know what you think, and your test results.