S3 Integration with ApigeeX

[marieorourke]

Is there an equivalent to the Amazon S3 Extension (ExtensionCallout policy) in ApigeeX?

If not, is there any other way to achieve the downloadFile Action with the content being returned as a JSON?

[dchiesa1 · Accepted solution]

There is not an "out of the box" extension policy in Apigee X that lets you call AWS S3. 

If you want to call to S3, you can do it with ServiceCallout, using the published API for S3.  The only trick there is the authentication - AWS defines a signature protocol for authentication, a cascaded HMAC, with maybe 4 levels if I recall correctly. This can be a little difficult to produce using the HMAC policy in Apigee, or with the hmac static function available in AssignMessage.  

Fortunately there is a community-contributed Java callout that can produce the AWS v4 signature for you, in one step. Find it here. The README for the callout includes this example for how you might configure it for use with S3: 

<JavaCallout name="JC-AWSSignV4">
  <Properties>
    <Property name="service">s3</Property>
    <Property name="endpoint">https://my-bucket-name.s3.amazonaws.com</Property>
    <Property name="region">us-west-1</Property>
    <Property name="key">{private.aws-key}</Property>
    <Property name="secret">{private.aws-secret-key}</Property>
    <Property name="source">outgoingAwsMessage</Property>
    <Property name="sign-content-sha256">true</Property> <!-- optional -->
  </Properties>
  <ClassName>com.google.apigee.callouts.AWSV4Signature</ClassName>
  <ResourceURL>java://apigee-callout-awsv4sig-20230705.jar</ResourceURL>
</JavaCallout>

But this callout does not call S3 for you.  A typical use of this callout involves 3 policies in order:

• AssignMessage - to create a message that will be sent to AWS (in your case S3)
• this callout - to compute the signature for the message, and set it into the Authorization header, and maybe set some other headers (like X-amz-date) 
• ServiceCallout - to send the message

[marieorourke]

Thank you Soo much.
I had found myself on that route, so delighted to hear i'm going the right direction.

[marieorourke]

Hi @dchiesa1 

When i implemented this solution, i am getting the following error on our Bamboo deploy

revision 29 has failed to deploy to dev-eu-2. 
Error: instance "europe-west2" reported error steps.javacallout.JavaCalloutInstantiationFailed: 
"Failed to instantiate the JavaCallout Class com.google.apigee.callouts.AWSV4Signature"

my policy looks like this

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<JavaCallout name="JC-AWSSignV4">
    <Properties>
        <Property name="service">S3</Property>
        <Property name="endpoint">https://s3.eu-west-1.amazonaws.com</Property>
        <Property name="region">eu-west-1</Property>
        <Property name="key">
            <Value ref="private.key"/>
        </Property>
        <Property name="secret">
            <Value ref="private.secret"/>
        </Property>
        <Property name="source">outgoingAwsMessage</Property>
        <Property name="sign-content-sha256">true</Property>
    </Properties>
    <ClassName>com.google.apigee.callouts.AWSV4Signature</ClassName>
    <ResourceURL>java://apigee-callout-awsv4sig-20230705.jar</ResourceURL>
</JavaCallout>

I have double checked the Artifact being deployed and i see the Jar is included and it doesn't appears to be corrupted.

Any ideas what i'm doing wrong?

[dchiesa1]

I think your policy needs to be like this:

<JavaCallout name="JC-AWSSignV4">
    <Properties>
        <Property name="service">S3</Property>
        <Property name="endpoint">https://s3.eu-west-1.amazonaws.com</Property>
        <Property name="region">eu-west-1</Property>
        <Property name="key">{private.key}</Property>
        <Property name="secret">{private.secret}</Property>
        <Property name="source">outgoingAwsMessage</Property>
        <Property name="sign-content-sha256">true</Property>
    </Properties>
    <ClassName>com.google.apigee.callouts.AWSV4Signature</ClassName>
    <ResourceURL>java://apigee-callout-awsv4sig-20230705.jar</ResourceURL>
</JavaCallout>

Not - private.key and private.secret are wrapped in curlies. But I also think the formatting of these Properties wouldn't cause the problem you are seeing: the failure to instantiate the class. Usually that happens because of a missing jar or a corrupted jar.  But it looks like you've checked that, and verified that the jar is good. 

For your information here's my jar

$ tar tvf ../bundle/apiproxy/resources/java/apigee-callout-awsv4sig-20230705.jar 
drwxr-xr-x  0 0      0           0 Sep 20 10:19 META-INF/
-rw-r--r--  0 0      0          82 Sep 20 10:19 META-INF/MANIFEST.MF
drwxr-xr-x  0 0      0           0 Sep 20 10:19 com/
drwxr-xr-x  0 0      0           0 Sep 20 10:19 com/google/
drwxr-xr-x  0 0      0           0 Sep 20 10:19 com/google/apigee/
drwxr-xr-x  0 0      0           0 Sep 20 10:19 com/google/apigee/encoding/
drwxr-xr-x  0 0      0           0 Sep 20 10:19 com/google/apigee/callouts/
-rw-r--r--  0 0      0         497 Sep 20 10:19 com/google/apigee/encoding/Base32$DecodingException.class
-rw-r--r--  0 0      0        2024 Sep 20 10:19 com/google/apigee/encoding/Base16.class
-rw-r--r--  0 0      0        3976 Sep 20 10:19 com/google/apigee/encoding/Base32.class
-rw-r--r--  0 0      0        6042 Sep 20 10:19 com/google/apigee/callouts/SignatureCalloutBase.class
-rw-r--r--  0 0      0         620 Sep 20 10:19 com/google/apigee/callouts/AWSV4Signature$Canonicalized.class
-rw-r--r--  0 0      0        8439 Sep 20 10:19 com/google/apigee/callouts/AWSV4Signature.class
-rw-r--r--  0 0      0       11256 Sep 20 10:19 com/google/apigee/callouts/AWSV4Signature$SignConfiguration.class

and when I download from github,  and check the shasum, it's here:

$ shasum -a 256 apigee-callout-awsv4sig-20230705.jar 
9130d97c6a6c762b765597c3a8410ae30e0741432864c8d7c264a477937de799  apigee-callout-awsv4sig-20230705.jar

Maybe you are aware: If you rebuild on your own, the sha hash will be different.